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ABSTRACT 

This tnird volume of a four-volume evaluation of the 
1988-89 New York City School Community Education Program (also knovm 
as the Umbrella Program) comprises reports evaluating seven 
innovative staff development and five curriculum development 
projects. Evaluation sources included student preprogram and 
postprogram test outcomes, writing samples, teacher and student 
questionnaires, and the number of acceptances of participants into 
special high schools. Overall, the program was not as successful in 
meeting its stated objectives as in previous years. Each report 
contains a brief project overview, describes the research 
methodology, presents the findings, and provides recommendations for 
improvement. The following projects are evaluated: (1) Moving Ahead, 
designed to help inexperienced teachers improve their teaching 
skills; (2) Staff Development in Writing Instruction; (3) Adventures 
in Social Studies; (4) Reading Instructional Support Effort (RISE); 
(5) Arts in General Education (AGE); (6) Enrichment Program, K-9; (7) 
Mathematics Instructional Support Program; (8) Adventures in Science; 
(9) Early Childhood Language and Literacy Program; (10) Discovering 
Abilities and Improving Achievement; (11) Sum in One, designed to 
help teachers improve elementary education music instruction; and 
(12) Oral History Program for senior nigh school teachers. 
Statistical data are included o.i 12 tables. Each report also includes 
examples of evaluation instrun-ients. (FMW) 
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SCHOOL COMMUNITY ED';CATION PROGRAM 
IN NEW YORK CITY ^ 
EVALUATION SUMMARY 



BACKGROUND 

The School Community Education Program (also known as the 
Umbrella Program) , administered by the Division of Curriculum and 
Instruction, provides a variety of educational and training 
experiences to a ^ride range of participants, including pre-* 
schoolers and their parents, and elementary, intermediate, and 
high school students, teachers, and supervisors. The program 
consists of 37 different projects designed to provide innovative 
solutions to local educational and school proble: s. Ten projects 
provide basic skills, English as a Second Language, and computer 
literacy instruction; ten focus on social issues and 
environmental studies; seven offer sraff development workshops; 
five involve curriculum development, and three are designed for 
prekiiiiergarten children. The remaining projects provide 
participants with a variety of educational experiences. 



POPULATION SERVE D 

In 1988-89, the program served some 25,000 students, 
primarily elementary school pupils. In addition, the program 
served* 1,100 teachers and supervisors and 100 prekindergarten 
children, as well as neighborhood adults in the 32 conanunity 
school districts and selected high schools. Each project 
established different selection criteria for program 
participation. 



PROGRAM OBJECTIVES 

Although program objectives were designed for each specific 
project and therefore varied, most focused on increasing the 
competence of project participants through mastery of specific 
skills and abilities. Most objectives also set quantitative 
criteria to be met by a minimum percentage of participants for 
the program to be considered successful. 



*This summary is based on th3 final evaluation report of the 
School Community Education Program in New York City 1988-89, 
prepared by the Office of Research, Evaluation, and Assessment/ 
Instructional Support Evaluation Unit. 
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EVALUATION METHODOLOGY 



The evaluation of the program is based on a number of data 
sources: student performance outcomes on standardized or 
project -developed tests, pupil writing samples, teacher and 
student survey questionnaires, number of acceptances to special 
high schools, and review of five curriculxm documents. These 
manuals and lesson plans were sent to different units of the New 
York City Board of Education's Division of Curriculum and 
Instruction for evc.luation. Preprogram and postprogram test 
outcomes were compared to determine mean differences and, when 
appropriate, correlated t-tests and effect sizes were also 
computed to establish stal^istical significance and educational 
meaningfulness, respectively. The percentage of participants 
^meeting quantitative project-set criteria for success was also 
determined . 



FINDINGS 

The 1983-89 evaluation findings indicate that the School 
Community Education Program was not as successful as it had been 
in previous years. Only 15 projects met their stated objectives, 
compared to 19 in 1987-88. In general, those projects providing 
staff development training and curriculum developmeiit were the 
most successful, in addition, two projects that provide remedial 
Instruction (Harleiii School-community Tutorial Project, and 
' Mathematics Improvement Program) were also found to be 
particularly successful. The evaluation also showed that 
although some projects met their objectives, these results should 
be treated with caution because of the vagueness of the 
objectives or because the evaluation instruments could not 
adequately measure project impact. This is a particular problem 
shared by staff development projects that seek to measure teacher 
ability to implement specific teaching skills in the classroom 
without including instruments which measure these skills. 

Four projects were successful in meeting one of their 
objectives, yet unsuccessful in meeting a second objective. 
Sixteen projects did not meet their evaluation objectives, and 
two projects could not be evaluated because test data were 
lacking. As indicated in previous years • evaluations, a few of 
these projects need extensive modifications, such as revision of 
testing instruments to avoid ceiling effect, development of 
project activities appropriate for different grade levels, or 
establishment of more stringent participant selection criteria. 
Most of the unsuccessful projectL,, however, failed to meet their 
objectives because their criteria for success were too stringent 
or because the testing instrument could not adequately measure 
project objectives, in some of these projects, participants 
achieved large mean gains, but the percentage of successful 
participants remained below the percentage established in the 
project-set criterion for success. In some cases, this criterion 
was beyond what could be reasonably expected of program 
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participants. 



RECOMMENDATIONS 

In addition, to the reconanendations made for each project, 
the following suggestions are made for the overall improvement of 
the School Community Education Program: 

* Closely monitor those projects that fail to meet their 
stated objectives. 

* Assist project staff in making necessary project 
modifications such as the revision of project activities, 
revision or replacement of testing instruments, 
establishment of adequate selection criteria of 
participants, or amendments in project objectives. 
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INTRODUCTION 

In 1988-89; the New York City Public Schools received 
$2,375,000 in funding from the New York State Legislature to 
operate the School Community Education program (also known as the 
Umbrella program) . It consisted of 37 different projects 
designed to provide innovative solutions to local educational and 
school programs. 

The program provided services to about 25,000 participants 
in 32 community school districts and selected high schools. 
While most of these participants were elementary school students, 
the program also served some 1,000 intermediate and high school 
students, 100 preschool children, and 1,100 teachers and 
supervisors. Some projects also included parenting components 
and/or sought to involve the parents of participating students in 
project activities. 

Evaluation reports are presented in four volumes. Volume I 
contains evaluations of ten projects that provided reading, 
mathematics, writing, English as a Second Language, and computer 
literacy instruction. Volume II includes evaluations of nine 
projects on social, ethnic, and environmental studies. Four of 
these projects also provided staff development workshops. Volume 
III contains evaluations of seven staff development and five 
curriculum development projects. The remaining six projects, 
presented in Volume IV, offered a variety of educational 
experiences to p^irticipants. Three of these projects were 
designed for prekindergarten children, and the other three 
projects were designed to teach students health maintenance 
concepts, to improve their acceptance rate to special high 
schools, and foster career awar';ness among students. 

Each report contains a brief project overview, describv^s the 
evaluation methodology, presents the findings, and provides 
recommendations for improvement. The reports are listed in order 
of budgeted function nuraber in the Table of Contents. 
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MOVING AHEAD, 1988-1989 

School -cor munity Education Program 
Program Adr-M,istrator: M. Morris Speiser 
Project Coordinator: Carol Williaios 

Prepared by: 
Office of Research, Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New rork City Public Schools 

PROJECT DESCRIPTIO N 

The purpose of .v^aving Ahead is to support and assist 
inexperienced teach'ii-s in Community School District (C.S.D.) 16 
to improve their tj.^-'jhing skills. The program's goal is to 
improve teachers' ^nanagement, instructional, and human relations 
skills through after-school courses and in-class support. 
Emphasis is placed on skills needed for the organization and 
management of an effective classroom. The New York State 
Legislature contributed $14 thousand in funding for this program. 
About 40 teachers participated in the 1988-89 program. 
/V Teachers were chosen by district supervisors based on their 

teaching experience. Participants in the program received 
college credit for after school courses. The after school 
courses, designed by C.S.D. 16, local college faculty, and the 
Staff Division of Curriculum and Instruction, emphasized school 
curriculum and management. Course activities included the design 
and use of appropriate materials, and the teaching of human 
relations skills designed to promote a greater understanding of 
students' diverse racial and cultural backgrounds. Follow-up 
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in-class assistance in the r.mplementation of skills learned in 
the courses was provided by the participating college 
instructors, District 16 supervisory personnel, and^ staff from 
the Division of Curriculum and Instruction. 

The objective for the program was for 80 percent of the 
teachers to demonstrate a 30 percent improvement in pedagogical 
skills as measured by a district-developed survey. 
EVALUi^TION METHODOLOGY AND FINDINGS 

The evaluation of the Moving Ahead program was based on a 
district-developed observational survey of effective pedagogical 
strategies (see Appendix A) . supervisory staff from the 
Community School District and the Board of Education provided 
follow-up monitoring of participants in their classrooms, 
utilizing the staff -developed survey to assess the teachers • 
pedagogical skills. 

An evaluation of this year's program is not possible, as no 
data was received by the Office of Research Evaluation and 
Assessment . 

CONCLUSIONS AND RECOMMENDATIONS 

Evaluation of this year's program was not possible because 
of the lack data. In 1987-88, the program was also not able to 
be evaluated for lack of pretest scores and incomplete reporting 
of posttest scores. It is recommended that, in tha future, 
greater efforts be made to insure that all pretest and posttest 
scores are available for evaluation. 

2 
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RATING SCALE FOR MOVING AHEAD 7iPPENDIX A 

B/E ^f5001-48-83403 93403 

Directions ; Please use the scale below to rate the teacher. 

Place the number that represents the observed 
Denavior on each blank. 

The Scale 

Above 

Poor Fair Average Average Excellent 

1 2 3 4 5 



Interacts with children in ways to encourage them to 
communicate thoughts and feelings verbally. 

Provides materials and activities to promote language 
development. 

Uses books and stories with children to motivate listening 
and speaking. 



Cognitive 



Helps children develop understanding of unfamiliar 
concepts and ideas. 

Interacts with children in ways which encourage them to 
think and solve problems. 

Provides games (instruction and direction games / memory 
games) . 

Provlaes tor expereinces to develop vocabulary. 

Provides classes atmosphere which is free from tension and 
which fosters social and intellectual growth. 

Provides first hand experience (objects, trips). 

Provides experiences for translating through and action 
into words (experience charts, activity charts, taking 
dicatation for children's stories, printing captions for 
children's drawings). 

Proviaes many opportunities for language skills 
development (speaking and sharing experiences, picture 
interpretations, storytelling , dramatizations, etc.) 




%3403 



Provides auditory discrimination exercises (e.g. dif- 
ferences in words f hearing sounds at beginning, middle 
and ending of words , rhyming words, hearing number of 
syllables in words) 



Provides visual discrimination exercises (e.g. 
recognizing colors, sizes, shapes, likenesses, differ- 
ences, left, right, picture clues) 



Word Recocnition Skills 



Eelps children build sight vocabularies (e.g. say 
words, see words and associate sounds with printed 
forms) 



Kelps children use contextual clues (determine word 
definition from sense of passage) 

Helps children use phonic analysis (e.g. recognize and 
say consonant sounds, vowel sound, understand simple 
phonicji generalizations) 



Helps children use structural analysis (e.g. recognize 
words from root parts and understand how changes added 
to or subtracted from the root affect meaning) 

Exercise related to areas such as: 

Compound words 

Inflectional endings 

comparisons: er, est. 

Prefixes 



Suffixes 



Syllabication 



Comprehension 



Kelps children get the main idea of a passag^^ 
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Helps children make inferences, read "between the 
lines" 



Kelps children note details 

Helps children make judgements and anticipate outccmes 
Helps children follow sequences of events 



/ 
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STAFF DEVELOPMENT IN WRITING INSTRUCTION, 1988-8? 

School-Coimnunity Education Program 
Program Administrator: M, Morris Speiser 
Project Coordinator: Shelley Harwayne 

Prepared by: 
Office of Research, Evaluation, and Assessmunt 
Instructional Support Zvaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Staff Development in Writing Instruction project 
provides training in the teaching of writing to elementary school 
teachers in 20 Community School Districts (C.S.D.s): 3, 5, 6, 7, 
8, 9, 10, 11, 14, 15, 17, 18, 20, 21, 22, 25, 26, 28, 29, and 30. 
District superintendents, school principals, and district 
curriculum specialists selected schools in each district to 
participate in the program. In a few of these schools, some 
teachers had already been trained in the program; in others, 
teachers were selected from among a group of volunteers who 
showed interest in improving their instructional skills in the 
wrii:ing process. 

In 1988-89, 600 teachers and 12,000 students participated in 
^the project. Teachers and their supervisors attended conferences 
and demonstration lessons conducted by two teacher-trainers and 
visiting consultants in topic^ such as writing as a process, 
improving writing through personal narrative, revision skills and 
techniques, and holistic scoring methods* Various instr^actional 
techniques were used, including teacher modeling, conferencing 
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between teachers and students, and presentations. The project 
objective was for students of the participating teachers to 
achieve a statistically significant and educationally meaningful 
mean gain from pretest to posttest, as measured by holistically- 
scored writing samples* The New York State Legislature provided 
$399,000 in funding for the purchase of equipment to support 
computer word processing and printing activities, and for 
classroom supplies. Funds were also used to cover the salaries 
of teachers' aides and consultants, and to involve teachers in 
after-school training activities. 
EVALUATION METHODOLOGY 

The evaluation of the project focused on analyses of scores 
of students' holistically-scored writing samples which reflected 

instructional skills of the participating teachers. To 
assess the program objective, students were given a writing 
assignment on the first and last day of project activities. 
Project staff selected a representative sample of students' 
writing assignments (20 per grade across the 20 participating 
districts for grades kindergarten through seven) . The writing 
samples were then holistically scored by a team of five raters 
who used scales developed by project staff. Each of the five 
raters reviewed and scored all of the randomly selected writing 
samples. For kindergartners and first graders, the raters used a 
scale drawn from the lists of stages children go through in 
learning to write that was developed by Marie Clay (see Appendix 

2 
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A) . For pupils in grades two through seven, the raters used a 
scale adapted from the Personal Narrative Writing Scale in Cooper 
and ^dell, Evaluating Writing (See Appendix B) . In both cases, 
students* writing samples were scored on a scale from 1 to 5. 

Pre- and post program holistic scores were submitted for 
evaluation. These scores were compared, and correlated t-tests 
were computed to establish if achievement differences were 
statistically significant. Effect size (E.S.),* which indicates 
the educational meaningfulness of the mean gain or loss for each 
comparison, was also calculated. 
FINDINGS 

Complete test scores were reported for 160 students. Table 
1 presents the students' pre- and posttest holistic scores by 
grade. Overall, students demonstrated a posttest mean gain of 
0.7 score points. This mean gain was found to be statistically 
significant and educationally meaningful. By grade, kindergarten 
students achieved the highest mean gain (1.1 score points), 
whereas seventh grade students achieved the lowest mean gain 



*The effect size, developed by Jacob Cohen, is the ratio of the 
mean gain to the standard deviation of the gain. The ratio 
provides an index of improvement in standard deviation units 
irrespective of the size of the sample. According to Cohen, 0.2 
is a small E.S., 0.5 is a moderate E.S., and 0.8 is considered to 
be educationally meaningful, reflecting the importance of the 
gains to the students' educational development. 

3 
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TABLE 1 



Students' Mean Pre- and Post-Program Holistic Scores" on 
Writing Samples, by Grade 
Staff Development in Writing Instruction, 1988-89 



Grade 


N 


Pre- 
Mean 


-Proaram 
S.D. 


Post- 
Mean 


■"Proaram 
S.D. 


Difference'' 
Mean S.D. 


E.S. 


K 


20 


1.9 


0.8 


3.0 


0.6 


1.1 


0.5 


2.2 


1 


20 


2.2 


0.7 


3.1 


0.8 


0.9 


0.7 


1.3 


2 


20 


2.5 


0.8 


3.2 


0.8 


0.7 


0.5 


1.4 


3 


20 


2.3 


0.9 


3.2 


0.6 


0.9 


0.7 


1.3 


4 


20 


2.3 


0.7 


2.9 


0.5 


0.6 


0.4 


1.5 


5 


20 


2.4 


0.8 


3.0 


0.6 


0.6 


0.7 


0.9 


6 


20 


2.7 


0.8 


3.2 


0.7 


0.5 


0.6 


0.8 


7 


20 


2.3 


0.7 


2.7 


0.6 


0.3 


0.5 


0.6 


Total 


160 


2.3 


0.8 


3.0 


0.7 


0.7 


0.6 


1.2 



^Based on scales from 1 (low) to 5 (high). 



All mean gains were statistically significant at p<.05. 

Students in all grade levels achieved mean gains 
ranging from 0.3 score points to !•! score points. 
These gains were statistically significant and 
educationally meaningful. 

Kindergarten students achieved the largest mean gains. 



4 



19 



93404 

(0,3 score points). Mean gains for grades kindergarten through 
SIX were statistically significant and educationally meaningful. 
The mean gain for grade seven, although statistically 
significant, showed only a moderate effect size, 
CONCLUSIONS AND RECOMMENDATIONS 

In 1988-89, the Staff Development in Writing Instruction 
project met its objective. Overall students' mean gain was 
statically significant and educationally meaningful. The 
evaluation findings indicate that the project had a significant 
and meaningful impact on participating teachers who, in turn, 
seemed to have influenced /the writing ability of their students • 

Although the program was successful, there was a variation 
in scores across the grades. Kindergarten through third grade 
students displayed the largest mean gains, while fourth through 
seventh grade students showed the smallest mean gains. The mean 
gain for grade seven, although statistically significant, showed 
only a moderated effect size. 

There are many possible explanations for this variation 
across grades. One explanation is the holistic scoring 
procedures utilized. Since children in the lov;er grades are just 
beginning to learn to write, it is impossible to use the same 
scoring critique for these children as that which is used for 
students of the higher grades. As a result, the high mean gains 
achieved by kindergarten and first grade students this year and 
in previous years may be a function of the different scoring 

5 
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criteria utilized. 

Another explanation for the variation of scores is the 
content of the instruction provided. Instruction must be 
tailored for each grade, since children of different ages have 
different skills and potentials. This makes it difficult to 
assure that each grade is recieving comparable instruction and, 
as a result, difficult to make any meaingful comparison across 
grades. 

In addition, it remains difficult to determine whether 
student growth can be solely attributed to the project's impact 
on participating teachers or whether it just reflects average 
student growth in the regular classroom. In the future, project 
staff may wish to evaluate the instructional skills of teacher 
participants by comparing the writing ability of students whose 
teachers were, trained by the Staff Development in Writing project 
with the performance of a control group of similar students whose 
teachers were not trained by the project. 
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DIRECTIONS FOR ADMINISTERING WRITING SAMPLE 



APPENDIX A 
93404 



PLEASE GIVE EACH CHILD A WRITING SAMPLE FORM AND A PENCIL (ONLY) o 
HELP THEM FILL OUT THE HEADING. (IF NECESSARY, FILL OUT THE HEADING 
FOR THEM BEFOREHAND) . ASK CHILDREN TO DRAW A PICTURE (WITH THEIR 
PENCIL ONLY) OF SOMETHING THEY LIKE TO DO* 

AFTER FIVE MINUTES/ ASK THEM TO TURN THE PAPER OVER AND WRITE OR 
PRETEND TO WRITE ABOUT THEIR PICTURE. DON'T LEAD THE CHILDREN INTO 
WRITING EXCEPT TO TELL THEM, "JUST PUT DOWN WHATEVER YOU CAN" OR 
"JUST TRY IT." REPEAT THE DIRECTIONS IF NECESSARY. 

★ NOTE ; IT SEEi^S APPROPRIATE AT THIS TIME OF YEAR THAT FIRST GRADERS 
BE ALLOWED TO USE LIKED PAPER. (ATTACHE PAPER TO FORM), 

TELL STUDENTS, "YCU ARE INVITED TO TAKE PART IN A SPECIAL WRITING 
ACTIVITY. PLEASE WRITE A TRUE STORY ABOUT SOMETHING THAT HAS 
HAPPENED TO YOU.'* THEN PROVIDE EACH STUDENT WITH TWO (2) SHEETS OF 
PAPER. SUGGEST THEY TAKE TIME TO LIST POSSIBLE TOPICS. ONCE THEY 
HAVE SELECTED ONE, THEY CAN BEGIN TO WRITE. LET THEM KNOW THEY CAN 
USE AS MUCK PAPER AS THEY NEED, AND THAT THEY HAVE PLENTY OF TIME. 
ALSO SAY, "YOU CAN TRY WRITING IT IN ROUGH DRAFT OR YOU CAN JUST 
START WRITING THE FINAL PIECE." 

IF SOME CHILDREN FINISH EARLY, ASK THEM TO READ A BOOK QUIETLY WHILE 
THE OTHERS WORK. AFTER 20 MINUTES, INTERRUPT THE CHILDREN WHO ARE 
STILL WRITING. SAY TO ALL CHILDREN, "IF YOU HAD MORE TIME TO WORK 
ON THIS PIECE OF WRITING AND YOU WANTED TO MAKE IT INTO THE BEST 
THAT IT COULD BE, WHAT WOULD YOU DO NEXT?" ON ANOTHER SHEET OF 
PAPER ASK THEM TO WRITE WHAT THEY WOULD DO NEXT. GIVE THEM FOUR 
(4) MINUTES TO DO THIS. 
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EVALUATION INSTRUMENT 

THE WRITING PROCESS PROJECT 

TEACHERS COLLEGE/UMBRELLA PROGRAMS 
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BECAUSE THE ULTIMATE GOAL OF THIS PROJECT IS TO IMPROVE THE QUALITY 
OF STUDENT WRITING, WE WILL HOLISTICALLY EVALUATE SAMPLES OF WRITING 
FROM THE FIRST AND LAST DAY OF OUR WORK IN A DISTRICT IN ORDER TO 
DETERMINE WHETHER THERE HAS BEEN SIGNIFICANT IMPROVEMENT IN THE 
WRITING. ON EACH OF THESE DAYS THE ASSIGNMENT WILL BE THE SAME. 
STUDENTS WILL BE ASKED TO SELECT A TOPIC OF PERSONAL SIGNIFICANCE TO 
THEM AND TO DRAFT AND REVISE THEIR PEICE WITHOUT INPUT FROM THE 
TEACHER. THE PIECES WILL BE DATED, SAVED AND THEN EVALUATED 
HOLISTICALLY. THESE DATA WILL BE GATHERED FROM AT LEAST TEN 
CLASSROOMS IN EACH OF THE 14 DISTRICTS AND THESE CLASSROOMS WILL BE 
REPRESENTATIVE OF THE GRADE LEVELS INVOLVED IN THE PROGRAM FROM THAT 
DISTRICT. WE WILL RANDOMLY SELECT TEN PERCENT OF THE CHILDREN FROM 
EACH 0\ THESE 140+ CLASSROOMS AND THE PRE- AND POST-SCORES WILL BE 
TABUL;iTED for those children. the goal, THEN IS TO HAVE A 
REPRESENTA1IVE SAMPLE ILLUSTRATING THE RANGE OF DISTRICTS AND GRADE 
LEVELS INVOLVED. 

METHODS FOR HOLISTIC EVALUATION WILL DIFFER SOMEWHAT DEPENDING ON 
THE AGE GROUP OF THE YOUNGSTER. TO THE BEST OF OUR KNOWLEDGE, NO 
ONE HAS ATTEMPTED TO EVALUATE THE EARLY WRITING OF PRIMARY CHILDREN 
IN THIS MANNER AND THEREFORE, WE HAVE DEVISED OUR OWN METHODS, WHICH 
ARE EXPLAINED LATER. WHEN THE STUDENTS ARE IN GRADES 2-8, 
HOWEVER, WE CAN DRAW ON AND ADAPT METHODS DESCRIBED IN COOPER'S 
TEXT, EVALUATING WRITING. 

A GROUP OF FIVE RATERS WILL EACH RANK ALL OF THE WRITTEN PIECES. 
THE RATERS WiLL ACHIEVE REALIABILTIY BECAUSE 1) THEY COME FROM 
SIMILAR BACKGROUNDS AND 2) THEY WILL BE CAREFULLY TRAINED TO REACH 
NEARLY PERFECT AGREEMENT ON SAMPLES USED FOR TRAINING PURPOSES. THE 
RATERS WILL EACH BE A PUBLISHED WRITER AND THEY WILL EACH HAVE A 
BACKGROUND IN TEACHING WRITING. AS COOPER SUGGESTS, THE RATERS WILL 
NOT USED THEIR IMAGE CF IDEAL PROFESSIONAL WRITING AS AN ABSOLUTE 
STANDARD OF QUALITY, BUT WILL INSTEAD RATE PAPERS RELATIVELY 
ACCORDING TO THE RANGE OF PAPERS PRODUCED. 

BECAUSE THE PIECES CF WRITIVG WILL BE PERSONAL NARRATIVES, THE 
RATERS WILL FOLLOW AN ADAPTATION OF THE GUIDELINES FROM THE PERSONAL 
NARRATIVE WRITING SCALE ON PAGES 21 - 24 IN COOPER'S TEXT (SEE 
ATTACHED ITEMS) . 

MANY OF OUR PRIMARY SCHOOL STUDENTS WILL NOT BE ABLE TO WRITE AT ALL 
AT THE START OF OUR TRAINING EFFORTS AND SO THEIR GROWTH WILL NEED 
TO BE EVALUATED ACCORDING TO DEVELOPMENTAL SEQUENCES OF EARLY 
WRITING. AGAIN, A TEAM OF FIVE RATERS WILL HOLISTICALLY EVALUATE 
THE SAMPLES AND AGAIN THE RATERS WILL PRACTICE THESE EVALUATIONS 
WITH SAi'lPLES OF WRITING SO AS TO ACHEIVE RELIABILITY. THE GUIDING 
SCALE, HOWEVER, WILL NOT BE THE PERSONAL NARRATIVE SCAlE BUT INSTEAD 
A LIST OF STAGES DRAWN FROM MARIE CLAY'S AND SUSAN SOWER'S 
DESCRIPTIONS OF THE STAGES CHILDREN GO THROUGH IN LEARNING TO WRITE 
(SHE ATTACHED ITEMS). WE \\1LL IDENTIFY THE STAGE EVIDENCED IN THE 
T:..^- and in THE POST-SAMPLES. THE RATERS FOR THESE PIECES WILL BE 
rZr.SONS trained in EARLY CHILDHOOD DEVELOPMENT AND ESPECIALLY IN 
TEACHING WRITING IN THE EARLY GRADES. 



23 



93404 

CHILDREN LEARNING TO WRITE (BASED ON ATTACKED LIST FROM WORLD 
RENOWNED RESEARCHER MARIE CLAY) 

A DEVELOPMENTAL LADDER 

LEVEL 1 - CHILDREN DO NOT APPEAR TO BE ABLE TO DIFFERENTIATE 
BETV;EEN a DRAWING AND WRITTEN LANGUAGF. 

LEVEL 2 - CHILDREN BEGIN TO USE SYMBOLS THAT ARE USED IN THE 
CULTURE'S SYSTEM OF WRITING. THEY WRITE STRINGS OF 
LETTERS OR SCATTERED LETTERS, BUT THERE APPEARS TO BE 
LITTLE SOUND-SYMBOL CONNECTION. 

LEVEL 3 - CHILDREN LABEL THEIR DRAWINGS OR WRITE WORDS ON THE PAGE, 
GENERALLY USING INITIAL AND FINAL CONSONANTS ONLY TO 
REPRESENT A WORD. 

LEVEL 4 - WORDS ARE COMBINED INTO SENTENCES AND SPELLINGS FILL OUT 
TO INCLUDE SOME MIDDLE CONSONANTS AND VOWELS, ALSO SOME 
SIGHT VOCABULARY. 

LEVEL 5 - CHILDREN USE THE WRITTEN CODE FOR A KIDE RANGE OF 

PURPOSES: LETTERS, POEMS, RECIPE BOOKS, SIGNS, ETC 
TKEY WRITE FLUENTLY. 
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CHILDREN LEARNING TO WRITE 



LEVEL 1 



LEVEL 2 



LEVEL 3 



LEVEL 4 



LEVEL 5 



LEVEL 6 



CHILDREN ARE NOT AELE TO DIFFERENTIATE BETWEEN A DRAaING 
AND WRITTEN LANGUAGE. 

CHILDREN ARE ABLE TO DIFFERENTIATE BETWEEN A DRAWING AND 
WRITTEN LANGUAGE. 

CHILDREN BEGIN TO USE SYMBOLS THAT ARE USED lU THE CULTURE'S 
SYSTEM OF WRITING. THE CHILDREN JUST WRITE STRINGS OF 
LETTERS, BUT WHEN ONE ASKS THEM TO READ WHAT THEY WROTE, THE 
CHILDREN GO ON AND ON. 

CHILDREN TRY TO CREATE AN ALTERNATIVE CORRESPONDENCE BETWEEN 
SPOKEN LANGUAGE AND WRITTEN LANGUAGE, E.G. THE WRITTEN 
RESPONSE MAY BE THE LENGTH OF THE SPOKEN UTTERANCE ACCORDING 
TO THE CHILD'S OWN REASONING. 

CKILD.REN BEGIN USING THE SYLLABIC HYPOTHESIS, I.E. USING ONE 
SYMBOL FOR ONE SYLLABLE. 

CHILDREN USE BOTH THE SYLLABIC AND ALPHABETIC HY^OT.^ESIS, 
(A RELATIONSHIP BETWEEN LETTERS AND SOUNDS) . 



LEVEL 7 - CHILDREN "BREAK CODE. " THEY ARE NOW ON THEIR WAY TO 

DEVELOPING THEIR WRITTEN LANGUAGE ACCORDING TO HOW ADULTS 
USE IT IN THE CULTURE. THIS IS WHEN THEY BEGIN TO GRAPPLE 
WITH THEIR INVENTED SPELLINGS AND BEGIN TO DISCOVER THE 
CONVENTIONAL SPELLINGS USED IN OUR ORTHOGRAPHY. 



A HIERARCHY OF SPELLING SKILLS: 

1. RANDOM STRING OF LETTERS 

2. BEGINNING SCUKDS ONLY 

3. BEGINNING AND ENDING SOUNDS 

4. BEGINNING, KIDDLE AND ENDING SOUNDS 



A TYPICAL PATTERN OF LEARNING THE LETTERS AND USING THEM: 

1. SINGLE CONSONANTS 

2. LONG VOWELS 

3. EVERYTHING ELSE IN NO SPECH.L ORDER: OTHER VOWEL SOUNDS, 
DIGRAPHS, CONSONANT BLENDS OR CLUSTERS. 



(SOWERS, SIX QUESTIONS TEACHERS 
ASK ABOUT INVENTED SPELLING) 



0:r 
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PERSONAL NARRATIVE WRITING SCALE 
(IX COOPER AND ODELL, EVALUATING WRITING, P. 21 



NOTE; FOR THE PURPOSES OF THIS EVALUATION, WE WILf SCORE ONLY 
THE FOLLOWING ASPECTS OF THE STUDENT WRITING: 

I. B. STYLE OR VOICE 

C. CENTRAL FIGURE 

E. SEQUENCE 

F . THEME 

II. A. AND B. WORDING AND SYNTAX 

D. AND E. PUNCTUATION AND SPELLING 
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#5001-48-93413 

ADVENTURES IN SOCIAL STUDIES; 1988-89 

School -Community Education Program 
Program Administrator: Morris Speiser 
Project Coordinator: John Paul Bianchi 

Joyce Rubin 

Prepared by: 
Office of Research/ Evaluation and Assessment 
Instructional Support and Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Adventures in Social Studies project sought to improv 
social studies instruction for grade six in Community School 
Districts (C.S.D.s) 8 and 18. The goal of the project was to 
train and a insist at least 32 teachers in the development and 
implementation of appropriate social studies lessons and 
activities. The New York State Legislature provided $22 thousand 
in funding for this project. 

Teacher part icipa"\">-- were selected by the district office 
based on their need for assistance in social studies instruction 
and on their interest in and ability to assist in the development 
of program activities. The training sessions were conducted at 
*"he district office and participating schools, and consisted of 
c*ftcr-school and in-school workshops conducted by the district 
social studies coordinator , social studies unit personnel, and 
other experts. 

The training sessions focused on the production of written 
instructional materials designed to strengthen the requirements 
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Of the New York State Regents Action Plan for Social Studies. On- 
site assistance including demonstration lessons, material 
selection, and the use and organization of classroom social 
studies centers was also provided to program participants. 

The program objective was for teacher participants to 
produce written instructional activities for use by seventh 
through ninth grade social studies teachets. The newly developed 
Instructional activities would include requirements indicated by 
the New York State Regents, the State Education Department, and 
the New York City Board of Education. 
EVALUATION METHODOLOGY ;^ND FINDINGS 

The evaluation of the Adventures in Social Studies project 
was based on a review of the completed documents through the use 
of an evaluation checklist (see Appendix A) designed to establish 
their effectiveness and to determine if they met the requirements 
of the New York State Regents, the State Education Department, 
and the New York City Board of Education. The document review 
for C.S.D. 8 was undertaken by the Director of Program and 
Curriculum Development and Instruction, and for C.S.D. 18, by the 
Director of the Social Studies Unit. 

The document produced by C.S.D. 8 was rated positively on 
all nine of the criteria of the evaluation checklist. The goals 
were found to be achievable and appropriate for fifth grade 
students and the activities were geared to the stated objectives. 

2 
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The doc ^ent produced by C.S»D, 18 was also rated positively 
on all nine of the criteria of the evaluation checklist. The 
reviewer stated that in general the lesson plans were well 
constructed and interesting, although he found that significant 
parts of the New York State syllabus and New York City curriculum 
were not addressed. He felt that the document could be 
distributed citywide as long as it was made clear that not all 
content recommended in the New York State Syllabus is covered^ 
CONCLUSIONS AND RECOMMENDATIONS 

The Adventures in Social Studies project for 1988-89 was 
successful. It produced written instructional activities for use 
by sixth grade social studies teachers that included requirements 
from by the New York State Regents, the State Education 
Department, and the New York City Board of Education. 

The document produced by C.S.D. 18 was rated positively on 
all criteria although it was stated that it did not address all 
information and requirements of the New York State Syllabus and 
the Regents Action Plan. In addition, the reviewer expressed 
concern about the "handout" materials utilized. He stated that 
the handouts were relevant, yet the quality of the reproductions 
were poor and citations were often incomplete or missing, in the 
future, program participants should attempt to cover all of the 
requirements of the New York State Regents, the State Education 
Department, and the New York City Board of Education and provide 
full documentation for all sources and "handouts" used. 

3 
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Citywide Ihfcrella Program ^^^^^S^^ ? 

E>/aluation Report for CurricuvVjon Projects Manuals 93413 
and Other Documents > (1988-39) 



Umbrella Program Name; Date: 



Name of |^«a»n Coirpleting the Review: 



Title: 

Introduction 

lbs State Education Department requires that all Uitorella Programs 
be evaluateed. In order to help us meet this requirement, we are asking 
that you examine this document, and evaluate it using this form, lhank you 
for your cooperation, 

Ihe manual follows the y Fj NA 

New York State syllabus and 
the New York City curriculum. 

Expla in : 



2. Ihe manual includes information Y N NA 

and requirements indicated by the 
Regents Action Plan. 

Expla in : 



3. Ihe manual int€3rates reasoning/ Y N NA 

thinking skills activities. 

Explain: » 



4, Ihe manual contains lesson plans Y N NA 

that present suitable strategies 
for achieving reasonable goals. 

Explain: 
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5, Ihe manual contains obDectives and 
concepts that are clearly defined. 

Explain: 



6. The nanual contains classroan 
activities and materials that 
are relevant and consistent with 
the stated objectives and teaching 
strategies, 

Expla in : . ■ 



7, The manual contains criterion 
referenced tests that include 
higher-level tninking questions. 



Explain:^ 



8. 



Ihe manual contains technical 
and non-technical language that is ^ 
consistent with the highest standaras 
of the Office of Professional Development 
and Leader snip Training. 



Explain:^ 



9; Tne manual coulo be circulatea citywide. Y 

Explain: 



10. The manual meets the goals specified in 
the objective of the original proposal. 



Explain: 
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READING INSTRUCTIONAL SUPPORT EFFORT (R.I.S.E.)i 1988-89 

School-Community Education Program 
Program Administrator: M. Morris Speiser 
Project Coordinator: Helen Guiliano 

Prepared by: 
Office of Research, Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Reading Instructional Support Effort (R.I.S.E.) is 
designed to provide staff development in the areas of reading 
assessment and remediation to teachers from C.S.D. 7. Through 
workshops and demonstration lecsons, the project seeks to train 
teachers to use a monitoring process and implement a 
developmental reading program. Activities include workshops in 
the implementation of alternative reading instructional 
activities, and training in the use and interpretation of various 
assessment instruments. In addition the program includes a 
parent workshop in which parents are encouraged to tnonitor their 
childrens' reading at home. 

In 1988-89, 71 teachers from nine elementary schools 
participated in the project. District staff and school 
principals selected teachers from schools that had overall 
reading achievement levels below grade level, Chapter 1 
designated and state designated Comprehensive Assessment Report 
(CAR) schools. Teachers attended after-school workshops twice a 
week in reading assessment and whole-language-reading process. 
After being trained, teachers worked with students after school 
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on an individual basis , teaching basic reading skills. 

The objective for 1988-89 was for 70 percent of the 
participating teachers to achieve a gain of 15 percent at 
posttest on a mastery checklist designed to measure teacher 
implementation of effective reading instructional activities 
prescribed by the program. In addition, 70 percent of the 
teacher participants were to achieve a rating of at least 75 
points on a district-developed survey designed to measure their 
attitudes towards the R.I.S.E. program. The New York State 
Legislature provided $26 thousf-nd in funding for the project. 
EVALUATION METHODOLOGY 

The impact of the program was measured through the use of a 
mastery checklist (see Appendix A) , designed to measure the 
teachers* effectiveness in implementing the instructional reading 
strategies taught in the workshops. Teachers received ratings on 
their effectiveness in mastering skills taught in the workshops 
in 15 different skill areas. The maximum possiLle score was 45 
•score points. Teachers were rated on the mastery checklist at the 
beginning and end of project activities. 

The teacher survey consisted of 20 items designed to measure 
participating teachers" attitudes toward the program (see 
Appendix B) . The survey included items on confidence in using 
R.I.S.E. materials, and use and availability of staff support. 
Maximum score possible on the survey was 100 points. 
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FINDINGS 

Complete test scores were submitted for 16 teachers (see 
Table 1). The overall mean pretest score was 28.6 points, or 
63.6 percent, and the mean posttest score was 41.4, or 92 
percent, for a mean gain of 12.8 points, or 28.4 percent.. 
Teachers at P.S. 30 showed the largest mean gain of 14.3 points, 
or 31.8 percent. One hundred percent of the teachers met or 
surpassed the project-set criterion of achieving a 15 percent 
gain at posttest. 

Data was submitted for 12 participants on the teachers 
survey (see Table 2) . The survey was broken down into five main 
sections. Overall mean rating on the survey for all items was 
67.6. Examination of the data by section reveals that the 
program received high ratings in the evaluation of the workshops 
and the confidence in and use of the materials provided. The 
program received lower ratings on the sections concerned with the 
availability and need of staff support. 
CONCLUSIONS AND RECOMMENDATIONS 

In 1988-89, the Reading Instructional Support Effort 
(R.I.S.E.) was success.ful in meeting one of its objectives, 
one hundred percent of participating teachers met or surpassed 
the project-set criterion of a 15 percent gain in the 
implementation of effective reading instructional activities. 
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Table 1 

Teachers' Mean Ratings on 
the Mastery Checklist*, by School 
Reading Instructional Support Effort, 1988-89 



Pretest Mean 
Raw Percent 
School N Score Correct 



Posttest Mean Mean Gain 

Raw Percent Raw Percent 
Score Correct Score Correct 



P-S. 5 4 22.5 50.0% 30.7 68.2% 8.2 18.2 

P.S. 30 12 30.7 68.2 45.0 100.0 14.3 31.8 

Total 16 28.6 63.6 41.4 92.0 12.8 28.4 

'Perfect raw score was 4 5 score points. 

One hundred percent of teacher participants met or 

surpassed the project-set criterion of a 15 percent 
increase on posttest. 
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Table 2 
Teachers' Mean Ratings 
on Project-Developed Survey* 
Reading Instructional Support Effort, 1988-89 





55pct i on* 




Maximum Score 
FoT Section 


Teachers • 
Mean Ratincfs 




Using the R.I.S.E. 
Proa ram 




20 


14. 3 




Evaluation of the 
R.I.S.E. Program 




15 


12.2 




Confidence and Ability 
in using Program 




20 


15.2 




Use of Resource 
Support 




25 


14. 5 




Identifying 
Future Needs 




20 


11.3 




Total 




100 


67.6 




"The survey was broken 
different content area 
12 teachers. 


into five main sections each dealing with 
The above mean ratings are based upon a 


a 

sample of 


Overall, teachers 


• mean 


rating of the project was 67.6 points. 


The section on Identifying Future needs received the lowest 
ratings. 


mean 
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As for the second objective, the data received by the Office 
of Research Evaluation and Assessment did not allow for a 
calculation of the number of individual teachers assigning a 
rating of 75 points. An examination of the low mean score would 
suggest that the objective of 70 percent of teachers assigning a 
rating of 75 points was not achieved. The Reading Instructional 
Support Effort was a new program for 1988-89, and as a result of 
the delay in the arrival of equipment, he program was not 
implemented as planned at all sights. This may be an explanation 
for the low ratings received on the teachers' survey. It also 
resulted in data being submitted for evaluation for only a small 
number of program participants. 

It is recommended that in the future, the teachers' survey 
be revised. The current survey contains a section on identifying 
future needs. This section received the lowest mean rating in 
1988-89. Although this type of information is useful for 
improving the program, it is not appropriate on a survey 
evaluating the current program. Questions on program improvement 
involve subjective opinions and desires, and are not useful in 
evaluating the effectiveness of the current program. In 
addition, some questions in other sections of the survey seem to 
be redundant and add no new information. 
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New York City 
Board of Education 



1 10 Livingston Street 
Brooklyn, New York 11201 



Dr. Richard R. Green 
Chancellor 



Office of Professional Oevelopmeni 
and Le a der ship Tnunini 
131 Livinpton Street 
Brooklyn. New York 1 1201 




Dr. Dolores M. Feniindez 
Deputy Chancellor 
tnstnicuon and Development 



Dr. Yveoe Jackson 

Director 
(718)933-4259 



R>I>S>E> testery Checklist 



Developed by the Office of Research, Evaluation Assessment 
Instructiaial Support for the Citywide Umbrella R.I.S.E. 

Program. 
#93401 (17) 



The Office of Research, Evaluation and Assessment has been asked to assess the 
inplementation and iirpact of the Umbrella R.I.S.E. Program. Because you work 
directly with the teachers, in your role as as teacher-trainer, we are asking 
you to complete the Mastery Checklist for each teacher participating the 
R.I.S.E. Program. 

Your input will be useful in describing how the program functions 
in a classroom. Thank you for your cooperation. 

Ihe Mastery Checklist must be conpleted ^twice for each teacher in the Progra:n; 
orKre at the beginn.lng, and again at the end of the program. Please use a 
separate form for each teacher. 



1) School: 2) Today's Date: 

3) Person completing this form: 

( teacher - tr a iner ) 

4) Name of Teacher:" 

5) l^acher's Current Grade Assignment: 

6) Length of time this person has been a classroom teacher: 
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Directions ; For each of the skills listed below, please circle th>2 "Level of 
Mastery" the teacher in question has addeved: Not Mastered, Partially 
Mastered, or Mastered, 

SKILLS TO BE MASTEIRED 



1. Using the Basal Reader materials 



to teach JReading: 



Not Mastered 



Partially Mastered 



Mastered 



2. Forming appropriate reading groups; 



Not Mastered 



Partially Mastered 



Mastered 



3» Managing multiple Basal Reader 
redding groups in the classroom: 



Not Mastered 



Partially Maatered 



Mastered 



4. Administering the RISE assessment 
iiQterials appropriately, at 
the end of each reading unit: 



Non Mastered 



Partially Mastered 



Mastered 



5. Understanding and interpreting 
the R.I.S.E. test results: 



Not Mastered 
Partially Mastered 
Mastered Not Mastered 



6. Redesigning an individual student's 
reading program, based on 
the RISE test results: 



Not Mastered 
Partially Mastered 
Mastered 
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7. Redesigning reading instructional 
programs for groups, based 
on R.I.S.E. test results: 


Not Mastered 
Partially Mastei 
Mastered 


8. Selecting and using alternative 
reading mater ials, beyond the Basal 
Reader series: 


h3ot Mastered 
Partially Mastered 
Mastered 


9. Offering students alternative 
reading mater ials, beyond the 
Basal Reader series; 


Not Mastered 
Partially Mastered 
Mastered 


10. Understanding how an individual 
child functions and develop? 
reading skills: 


Not Mastered 
Partially Mastered 
Mastered 


11, Understanding how a reading group 
functions and develops: 


Not Mastered 
Partially Mastered 
Mastered 


12. Managing the classroom and 
maintaining discipline: 


Not Mastered 
Partially Mastered 
Mastered 



ERJC 
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13. 


^iaking appropriate use of resource 
suDOorti* the teacher — tr a inpr 
program, and peer support: 


Not Wasiiered 
Mastered 






nCLJcpuiil^ \^£^xuxCaA XceOOaCi^! 


Not Mastered 
Partial?./ Mastered 
Mastered 




15. 


Maintaining an openness towards 
innovative reading naterialsr 
and an interest in new instructional 
aoor caches . 


Not Mastered 
Partially Mastered 
Mastered 






GENERAL COMMENTS AND OBSERVATIONS: 
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Office of Research, Evaluation and Assessment 

Instructional Sup ^ort Evaluation Unit 
Reading Instructional Support Effort, R.I.S.E. 
Teacher Survey #1, 1988-1989 



The Office of Research, Evaluation and Assessment has been asked 
to assess the implementation and impact of the R.I.S.E. Program* 
Your responses to the questions below will be useful in 
describing how this program evolved? what works, and how to make 
it more effective. 

We hope that you respond candidly and completely. Be assured 
that your responses are confidential. Thank you for your 
cooperation. 



Please write in your: School: Grade you teach: 

Years of Public School Teaching experience: 



Directions : Please circle the word or phrase that most closely 
corresponds to your response to each statement below. 

A. USING THE R.I.S.E. PROGRAM 

1. I used the RISE assessment materials at the end of each 
reading xinit, on a weekly basis: 

almost always often sometimes rarely not at all 

2* I used the RISE assessment materials at the end of each 
reading xinitr on a monthly basi^: 

almost always often sometimes rarely not at all 

3. I used the RISE test results to redesign my instructional 
program for students, or an individual basis: 

almost always often sometimes rarely not at all 

4. I used the RISE test results to redesign my instructional 
program for students, on a group basis: 

almost always often sometimes rarely not at all 



93417 



#17 Teacher Survey, pg.2 

B. EVALUATION OF THE R.I.S.E. PROGRAM 

5. The Assessment Materials matched the instructional objectives 
of the Basal Readers I use. 

strongly no firm strongly 

agree agree opinion disagree disagree 

6. The Assessment Materials increased my underscanding of how an 
individual child functions and develops reading skills. 

strongly no firm * strongly 

agree agree opinion disagree disagree 

7. The Assessment Materials increased my understanding of how a 
reading group functions and develops. 



strongly no firm strox^y 

agree agree opinion disagree disagree 

C. CONFIDENCE AND ABILITY USING THE R.I.S.E. MATERIALS 

How would you rate your confidence and ability in the following 
areas? 

8. Administering the RISE tests to students. 

^®^y somewhat very 

confident confident hesitant unsure unsure 

9. Understanding and interpreting the test results. 

very somewhat very 

confident confident hesitant unsure unsure 

10. Using the RISE test results to make changes in the reading 
program for individual students. 

vf^y somewhat very 

confident confident hesitant unsure unsure 

11. Using the RISE test results to change a group's reading 
program. 

• 

very somewhat very 

confident confident hesitant unsure unsure 
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#17 Teacher Survey, pg«3 

D. USE OF RESOURCE SUPPORT 

How often did you seek out Resource Support? 

12. To discuss classroom management issues: 

consistently often sometimes rarely not at all 
13* To use the Basal Readers: 

consistently often sometimes rarely not at all 

14. To administer the R.I.S.E. tests: 

consistently often sometimes rarely not at all 

15. To learn about alternative teaching strategies: 
consistently often sometimes rarely not at all 

16. To offer students alternative reading materials: 
consistentxy often sometimes rarely not at all 

E. IDENTIFYING FUTURE NEEDS 

How much support vould you like to receive in the following 
areas? 

17. Managing multiple Basal Reader groups in the classroom. ' 

a lot of some no firm a little no support 

support support opinion support necessary 

18. Organizing the R.I.S.E. assessment process: 

a lot of some no firm a little no support 

support support opinion support necessary 

19. Interpreting the R.I.S.E. assessment results: 

a lot of some no firm a little no support 

support support opinion support necessary 

20. Selecting and using alternative teaching strategies: 

a lot of some no firm a little no support 

support support opinion support necessary 

Are there other issues or areas of concern you would like the 
R.I.S.E. program to address? 
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ARTS IN GENERAL EDUCATION (AGE) , 1988-89 

School-Community Education Program 
Program Administrator: Morris Speiser 
Project Coordinator: Elton Warren 

Prepared by: 
Office of Research, Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Arts in General Education (AGE) project is designed to 
assist teachers in planning and integrating the arts into their 
general education classroom curriculum • The goal of the project 
is to improve the instructional skills of participating teachers 
so that they can provide more stimulating learning experiences 
for students. The AGE project received $37 thousand in funding 
from the New York State Legislature. 

In 1988-89, 98 elementary and high school teachers and 
supervisors from 11 Community School Districts (C.S.D^s 2, 6, 8, 
9, 10, 12, 15, 19, 22, 24, and 27) participated in the program. 
Teachers who indicated their willingness to participate in the 
project were selected by school principals. 

Participants attended a series of four staff development 
workshops and follow-up sessions which focused on dance as the 
catalyst for developing learning experiences in all the arts. 
Under the auspices of AGE/u ^ ^e Theater Foundation Dance 
Education Partnership, the workshops were conducted by 
specialists from the Limon Dance Company and the Merce Cunningham 
Dance Company. Instructional activities included demonstration 
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lessons, dance classes, and performances. Conferences were 
provided for principals to support staff development training. 
Students also attended dance performances at the Joyce Theater. 
The objective for 1988-89 was for participating teachers to 
demonstrate their ability to integrate knowledge regarding the 
arts into the basic instructional program. This was measured by 
three different instruments: workshop evaluation forms, completed 
by the participating teachers; teacher surveys, completed by 
teacher's supervisors; and principal surveys, completed by the 
principals of participating schools. 
EVALUATION METHODOLOGY 

Evaluation activities focused on three areas: teachers' 
responses to the AGE workshops; the workshops' impact on the 
instructional practices of teachers as assessed by their 
supervisors; and the impact of the program on the participating 
schools as assessed by the principals. A different instrument 
was developed to measure each objective. 

Teacher response was measured by a workshop evaluation i^orm 
distributed at each workshop (see Appendix A). The objective for 
this project component was for teachers to assign a mean rating 
of at least 20 points to each workshop. Teachers were asked to 
rate five statements about the workshop on a five-point scale, 
from "strongly agree" to "strongly disagree," and to describe the 
positive and negative features of the workshops. The highest 
possible score was 25 points. 

2 
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The impact of the workshops on the teachers' instructional 
practices was assessed by the teacher survey completed by the 
teachers supervisors (see Appendix B) . The survey consisted of 
five statemenvs about the teachers^ use of art activities in the 
classroom, and asked the supervisor to rate how frequently the 
statement was true of the teacher. The criterion for success was 
for teachers to receive a rating of at least 12 points on the 
teacher survey. To facilitate analysis, a numerical value was 
assigned to each response: never=0, seldom=l, sometimes=2r and 
frequently=3. The highest score possible was 15 points. 

The impact of the AGE program on the participating schools 
was essessed by a principals' survey (see Appendix C) . The 
objective for this component was for principals to give the 
project a rating of at least ten points. Principals were asked 
to respond to four questions rating the extent of project impact 
on teachers and students at their schools, and to cite one 
specific example of AGE's contribution to the school program. To 
facilitate analysis, a numerical value was assigned to sach 
response: no external program provided=0, no observable effect=l, 
somewhat=2, considerabiy=3 . The highest score possible was 12 
points. 
FINDINGS 

Complete data was submitted for only two teachers on the 
teachers' survey and two principals on the principals' survey. 
No survey data was submitted for the workshops. On the teachers' 

3 
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survey, both of the teachers rated the program positively on all 
points, overall mean rating was 20 out of a possible 20 points. 
On the principals' survey, the program again received positive 
ratings on all items with a mean of 11 out of a possible 12 
points (see Table 1) . 
CONCLUSIONS AND RECOMMENDATION? 

In 1988-89, an adequate evaluation of the Moving Ahead 
program was not possible because of the small number of surveys 
submitted to the Office of Research Evaluation and Assessment. 
Data was received for only two teachers on the teachers' survey 
and two principals on the principals' survey and no surveys were 
submitted for the workshops, of the surveys submitted for 
evaluation, the program received very high ratings from both 
teachers and principals. Principals stated that children were 
enthusiastic about that program. Considering the high ratings on 
the surveys received, including the principals' written-in 
comments, it is believed that this program is indeed having an 
impact on participants, in the future, project staff should make 
a greater effort to insure that all scores are submitted to the 
Office of Research, Evaluation, and Assessment for evaluation. 



4 
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Table 1 

Principals' Rating of Project Impact on Their Schools 
Arts in General Education, 1988-89 



Survey Item 



Mean Rating* 



Skills learned in AGE workshops 
benefited classroom instruction 
at school 



3.0 



AGE teachers shared their 
experiences with other teachers 
at school 

Teachers and/or students benefited 
from participating in AGE-sponsored 
external classroom special programs 

Teachers are interested in 
participating in future AGE training 
workshops 



Total' 



(N=2) 



2.0 



3.0 



3.0 



11.0 



•The following ratings were used; considerably=-3 , somewhat=2, no 
observable effect=l. 

^otal mean rating for all survey items. Highest possible, 
rating=12 . 

* Total mean rating for all items on the survey was 11.0 
points. 
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DIVISIOfJ OF CURRICULUM AND INSTRUCTION 
PEARL K. WARNER, EXECUTIVE DIRECTOR 



WORKSHOP EVALUATION FORM 

DEAR PARTICIPANT: YOUR IHPUT FROM THIS COMPLETED EVAUATICN FORM WILL ASSIST 
US TO MODIFY AND IMPROVE FUTURE WORKSHOPS. THANK YOU FOR YOUR COOPERATION. 

DISTRICT 

TITLE OF WORKSHOP: 

WORKSHOP LOCATION:^ 

WORKSHOP TITLE: 



WORKSHOP PARTICIPANT TITLE: 

SUPERVISOR (ADMINISTRATOR) DISTRICT STAFF 

OTHER (SPECIFY) 



TEACHER (GRADE LEVEL) 

PARAPROFESSIONAL 

SCHOOL (OPTIONAL) 



NAME OF RESPONDENT (OPTIONAL) 



STRONGO STftONSLY 
AGPEE AGREE UNDECIDED DISAGREE DISAGREE 
. 5 4 3 ? 1 

THE WORKSHOP WAS EFFECTIVE 
IN PRESENTING THE MATERIAL 
IN A MANNER TMAT WAS USE- 
FUL IN IMPROVING CLASSROOM 
INSTRUCTION. 

THE MATERIALS USED WERE 
RELEVANT AND APPROPRIATE 
TO THE TOPIC. 

THE GOALS AND OBJECTIVES 
FOR THE WORKSHOP WERE 
ACHIEVED. 



WORKSHOP PARTICIPANTS' 
QUESTIONS WERE ENCOUR- 
AGED AND ANSWERED 
PROFESSIONALLY. 



THE WORKSHOP DEMONSTRA- 
TION LEADER WHO WAS Kr) 
KNOWLEDGEABLE ABOUT THE 
Q PROGRAM. 

EMC 
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RECOMMENDATIOHS - GENERAL COMMENTS 

FIRST : PLEASE LIST FROM YOUR PRCFESSIOHAL PERCEPTIONS AND WORKSHOP 
EXPERIENCE THE THREE (3) MOST SIGNIFICANT POSITIVE FEATURES, QUALITIES 
AND/OR HIGHOIGHTS OF THE WORKSHOP YOU JUST PARTICIPATED IN. FIRST 
IHPRESSIOKS ARE IMPORTANT. ONE WORD OP A BRIEF SENTENCE TO DESCRIBE 
YOUR PRESENT FEELING IS ADEQUATE. 

POSITIVE FEATURES OF WORKSHOP ; 

1. 



2. 



3. 



SECOND: LIST ANY NEGATIVE FEELINGS YOU HAVE ABOUT THE WORKSHOP EXPERIENCE. 
~ IF- NONE, PLEASE WRITE NONE. THANK YOU. 

NEGATIVE FEELINGS ; 

1. 



2. 



3. 



USE THE BACK OF THIS SHEET FOR ANY ADDITIONAL RECOMMENDATIONS OR GENERAL 
COMMENTS. 



001 OF 

«^ Jl. 
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ARTS IH 6FHERAL EDUCATION (AGE) 
TEACHERS' SURVEY 



TEACHER'S NAME GRADE LEVEL 



SCHOOL SPECIAL SUBJECT AREA 



AGE PARTICIFAMTS (CIRCLE ONE): YEAR 1 YEAR 2 YEAR 3 



MORE 



1. THE TEACHER'S LESSOK PLANS INDICATE THAT SHE/HE INTEGRATES ARTS 
ACTIVITIES INTO CLASSROOM INSTRUCTION. "iti^KMiti, ARIb 

A- HEVER B. SELDOM C. SOMETIMES D. FREQUENTLY 

2. AGE TEACHERS SCHEDULED EXTERNAL ARTS ACTIVITIES FOR STUDENTS. 

A. NEVER C. SELDOM C. SOMETIMES D. FREQUENTLY 

3. THE TEACHER PROVIDES INSTRUCTION IN BASIC ART CONCEPTS. 

A- "EVER B. SELDOM C. SOMETIMES D. FREQUENTLY 

4. THE STUDENTS' BEHAVIOR DEMONSTRATES ENTHUSIASATIC RESPONSE TO 
ARTS ACTIVITIES. 

A- f-'EVER B. SELDOM C. SOMETIMES D. FREQUENTLY 

5. THE TEACHER ENCOURAGES STUDENTS' CREATIVE EFFORTS. 

A. NEVER 8. SELDOM C. SOMETIMES D. FREQUENTLY 
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AaTS IK GENERAL EDUCATION (AGE) 
PRINCIPALS' SURVEY 
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PRINCIPAL'S NAME 
SCHOOL 



1. .SKILLS LEARNED BY TEACHERS IN AGE TRAINING WORKSHOPS BENEFITED 
CLASSROOM INSTRUCTION IN MY SCHOOL. 

A. CONSIDERABLY B. SOMEWHAT C. NO OBSERVABLE EFFECT 



AGE TEACHERS SHARED THEIR EXPERIENCES WITH OTHER TEACHERS AT HY 
SCHOOL. 

A. CONSIDERABLY B. SOMEWHAT C. MO OBSERVABLE EFFECT 



3. TEACHERS AIJn/QR STUDENTS AT MY SCHOOL BENEFITED FROM 

PARTICIPATION IN AGE-SPONSORED EXTERNAL CLASSROOM SPECIAL 
PROGRAMS. 

A. CONSIDERABLY B. SOMEWHAT C. NO OBSERVABLE EFFECT 

D. HO EXTERNAL SPECIAL PROGRAM PROVIDED. 



4. TEACHERS AT MY SCHOOL ARE INTERESTED IN PARTICIPATING IK FUTURE 
AGE TRAINING WORKSHOPS. ' 

A. YES, THERE IS CONSIDERABLE INTEREST 

B. INTEREST IS LIMITED TO A FEW TEACHERS 

C. NO, TEACHERS DO NOT WANT TO PARTICIPATE 

5. CITE AT LEAST CHE SPECIFIC EXAMPLE OF HOW AGE PARTICIPATION 
CONTRIBUTED TO YOUR SCHOOL PROGRAM THIS YEAR. 



0018F 



#5001-48-93423 

ENRICHMENT PROGRAM K-9 , 1988-89 

School -Community Education Program 
Program Administrator: M. Morris Speiser 
Project Coordinator: Barbara Slatin 

Preparecl By: 
Office of Research; Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Enrichment Program 7-9 provides staff development 
workshops to elementary and intermediate school teachers in 
Community School Districts (C,S,D,s) 3, 11, 13, 19, 20, 21, 24, 
25, 26, 27, 28, and 29. This program is designed to motivate and 
train teachers to meet the needs of high-achieving students in 
kindergarten through grade nine. 

In 1988-89, 515 teachers were selected by their principals 
to participate in the project. They attended five all-day 
workshops focusing on such topics as the assessment of 
instructional needs of gifted students, Taylor's Multiple Talent 
Theory, and the Enrichment Renzulli Triad Model. The workshops 
were conducted by district staff and consultants expert in gifted 
education. These experts visited the classroom of each 
participant to provide assistance in implementing project 
activities. In addition, teachers and students from C.S.D. 13 # 
20, 24, and 29 were involved in artistic, dramatic, and museum 
activities designed to stimulate creative expression in students 
and enrich the curriculum offerings of the teachers. 

The objective for 1988-1989 was for 75 percent of the 
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participants to improve their knowledge of teaching techniques in 
the ar'O^Ls of instructional management, reasoning skills, and 
curriculum enrichment by at least 30 percent on a project- 
developed test. In addition, eighty percent of the ceachers 
involved in the artistic and museum activities were expected to 
assign a rating of at least 35 on a survey developed to measure 
the effectiveness of the activity in enriching classroom 
instruction. The project received $320 thousand in funding from 
the New York State Legislature. 
EVALUATION METHODOLOGY 

Project impact was asser.sed by an analysis of teachers' 
scores on a project-developed test (see Appendix A) and their 
ratings of the project museum activities on a project developed 
survey. The 30-item test measures knowledge of Taylor's Multiple 
Talent Theory, forms of thinking, ahd strategies for giftcl 
education. The test was administered on a pretest and posttest 
basis at the beginning and end of the program. The survey 
consisted of 10 items designed to measure the effectiveness of 
the museum activities in enriching classroom instruction (see 
Appendix B) . 
FINDINGS 

Complete test scores were reported for 6 teachers from 
C.S.D. 27 and 28 (see Table 1). Overall mean gain was 33.7 
percent. The pretest mean raw score for all districts was 8.-/ 
points (29 percent correct responses) and the posttest mean score 
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wai? 18.8 points (62.7 percent correct responses). Teachers at 
C.S.D. 28 received the largest mean gain of 10.5 raw score 
points, or 35 percent. 

Table 2 shows the percentage of participants who met the 
project-set criterion, by district. Overall, 92 percent of the 
participeting teachers improved their knowledge of teaching 
techniques by at least 30 percent. 

Two hundred and forty-nine teachers completed the survey 
rating the effectiveness of the museum and artistic activities on 
enriching their classroom curriculum. The overall mean rating 
was 41.5 out of a possible 50 points. Seven+'y-nine percent of 
the participating teachers assigned ratings ? 35 or over on the 
effectiveness of these activities. 
CONCLUSIONS AND RECOMMENDATIONS 

In 1988-89, the Enrichment Program K-9 was successful in 
meeting its objectives. Overall, ninety-two percent of the 
teachers who were administered the project-developed test 
improved their knowledge of teaching techniques in the areas of 
instructional management, reasoning skills, and curriculum 
enrichment by at least 30 percent. In addition, 79 percent of 
the participating teachers assigned ratings of 35 or more on the 
effectiveness of the project's artistic and museum activities on 
enriching classroom curriculum. 
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TABLE 1 



Teachers' Mean Raw Scores* on a Program-Developed Test 

By District 
Enrichment Program K-9, 1988-89 









Pretest Hjipfln 


Posttest M^^n 


Mean Gain 


District 


N 


Raw Percent 
Score correct 


Raw Percent 
Score Correct 


Raw Percent 
Score Correct 



27 


38 


9.0 


30.0 


18.9 


63.0 


9.9 


33.0 


28 


22 


8.3 


27.7 


18.8 


62.7 


10.5 


35.0 


TOTAL 


60 


8.7 


29.0 


18.8 


62.7 


10. 1 


33.7 



Perfect Raw Score=30. 



Overall mean gain was 33.7 percent points. 
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TABLE 2 

Percentage of Participants Meeting Project-Set Criterion* 

by District 
Enrichment Program K-9, 1988-89 



Meetincf criterion 



District 


N 


N 


% 


27 


38 


35 


92.1 


28 


22 


20 


90.9 


TOTAL 


60 


55 


91.7 



"Seventy percent of the participating teachers will have improved 
their knowledge of teaching techniques by at least 30 percent. 



* Ninety-two percent of the participants met the project-set 
criterion for success. 
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In 1987-88, complete pretest and posttest data for the 
project-developed test was received for 242 teachers from seven 
C.S.D.s. In 1988-89, complete test data was received for only 60 
teachers from C.S.D.s 27 and 28, although survey data was 
provided for 249 teachers from all participating districts. It 
is unclear whether this change in reported Jata reflects changes 
in the program's objective toward a greater concentration on the 
museum activities, or whether the data were simply under- 
reported. In the future, project staff should report pretest and 
posttest data for all teachers participating in the staff 
development component of the program. 



APPENDIX A 

ENRICHMENT PROGRAM K-9 PRE - POSTTEST 93423 



NAME DATE 

SCHOOL DISTRICT 



1) Calvin Taylor's approach to the teacher-learning process is called 
the; 

a) multiple talent approach, 

b) content process approach. 

c) product orientation method. 



2) In view of what you know of Taylor's Rationale, which of the 
statements listed below would best describe a talent implementation 
program in the classroom? 

a) separately from the acquisition of knowledge* 

b) simultaneously with the acquisition of knowledge. 

c) alternately with the acquisition. 

3) Select the component (s) which are incorporated in the complex 
process of the Multiple Talent Approach to learning: 

a) cognitive. 

b) affective. 

c) neither of these. 

d) both of these. 

4) If you had a class from a low-socio economic background, what could 
you expect of them in talent development? Choose the statement you 
feel is most accurate. 

a) some would be talented in all areas. 

b) given enough time 85% woula show achievement in several areas* 

c) 9 out of 10 employ at least one talent with above-average 
efficiency both for acquiring knowledge and for solving problems* 

T F 

5) When we speak of "gifted" students we are referring to 

a very homogeneous group of individuals. 

6) The individual intelligence test is che only true 

indicator of giftedness* 

7) The gifted program should be separate and independent 

of other school programs. 

8) The gifted program should be concerned with providing 

learning opportunities and experiences that will make 

up for deficiencies in the regular classroom. 



60 
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9) It is really important for the gifted program to have 

a separate and unique identity in your school. 

10) Divergent thinking is a type of thinking where there 

IS usually one answer. 

11) Remembering and recognizing information is the 

student's main job. 

12) The studen't job is to know the best answer to each 
problem. 



DIRECTIONS write the appropriate talent area for the following student 
behaviors on the blanks that precede each question. 

Productive Thinking 
Communication 
Forecasting 
Planning 
Decision Making 



13) The studeu has recorded his final choice for a career. 

14) The student is composing a poem about the joy of being 
an American citizen. 

15) The student is sharing problems he anticipates as he 
acts as a host to a friend for an afternoon. 

16) The student is comparing yellow flowers to many 
different other yellow things. 

^"^^ toy ^^"^^"^ brainstorming many ways to improve a 

18; The student is pantomiming how a banana feels beina 
peeled. 

19) The student is using many words to describe a rock for 
the science display. 

20) The student is adding details to his chalk picture of 
a duck waddling in peanut butter. 

21) The student is making different predictions as to what 
caused the car accident. 

22) The student is recording many different things mud is 
as swishy as. 

23) The student adds details t-, his special birthday gift 
ideas to make them even better. 
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24) The student is generating ideas for "one of a kind" 
birthday gifts. 

25) The student has made a final choice of one campaign 
strategy as being the best. 

26) The student is predicting the many different causes for the 
flat tire on his bicycle. 

27) The student is using a variety of single words that 
describe the shell he is observing after a field trip to 
the beach. 

28) The student is listing all of the materials and equipment 
he will need for his magic show. 

29) The student is using body language to demonstrate how to 
row a boat at the beach. 

30) The student is drawing and labeling many different that are 
as spiney as a starfish. 
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ANSWER KEY 





A 


16. 


Communication 


2. 


B 


17. 


Productive Thinking 


3. 


D 


18 - 


Communication 


4, 


C 


19. 


Communication 


5. 


FALSE 


20. 


Productive Thinking 


6. 




21. 


Forecasting 


7. 


FALSE 


22. 


Communication 


8 . 


FALSE 


23. 


Productive Thinking 


9 . 


FALSE 


24. 


Productive Thinking 


10. 


FALSE 


25. 


Decision Making 


11. 


FALSE 


26. 


Forecasting 


12 


PAT 


27 . 


Communication 


13. 


Decision Making 


28. 


Planning 


14. 


Communication 


29. 


Communica :ion 


15. 


Planning 


30. 


Communication 
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Survey for Umbrella Program 93^01 (#23) 



SCHOOL: 



GRADE: 



DISTRICT: 



OATF: 



Check if Appropriate: Special Education 



Bi 1 ingual 



!• TYPE OF ACTIVITY: 
(check one) 



A. Single Performance, Auditorium: 

B. Single Performance, Classroom: 

C. Series of Hands*on Workshops: 
D* Single Hands-on Workshop: 

£• Other (Specify) 



II. TEACHER EVALUATION: Please rate the activity you participated in by 
putting a check in the box which corresponds to your assessment of the 
program, using a scale from I*5* 



The Instructional 

/>W w 1 ▼ 1 ^ 7 • 

1* was age/grade appropriate* 


POOR 
1 


BELOW 
2 


AVERAGE 
3 


ABOVE 
Ai/PPAftr 

M VbltMUC 

k 


EXCELLENT 
5 


2. was well organized. 












3. challenged students' social 
and intellectual level* 












expanded students' appreciation 
of cultural diversity* 












5* was exciting and educationally 
stimulating* 












6* will help students learn new 
skills, or apply existing ones 
in new situations.- 












7. was relevant to my teaching 
curriculum. 












8. presented information and/or 
•ideas vhich I can integrate 
into everyday teaching of 
different subject areas. 












9. made efficient use of 
classroom time. 












10. was worthwhile, and T would 
recommend th?5; program to 
other teachers. 
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MATHEMATICS INSTRUCTIONAL SUPPORT PROGRAM, 1988-89 

School -Community Education Program 
Program Administrator: M. Morris Speiser 
Project Coordinator: Phyllis Gonon 

Prepared by: 
Office of Research, Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Mathematics Instructional Support System was organized 
to develop a sequentially designed, supplementary mathematics 
guide for grade five in Community School District (C.S.D,) 18. 
The written manual integrated reasoning and thinking skills 
activities into the grade five Board of Education Comprehensive 
Instructional Management System in Mathematics (CIMS-Math). The 
purpose of the project is to assist mathematics teachers in 
their effort to improve the reasoning abilities of students. 
Eighty teachers participated in the program in 1988-89. The New 
York State Legislature provided $18 thousand in funding for 
staff support, supplies, and materials. 

In 1988-89, a committee consisting of C.S.D. 18 teachers 
and District Office staff met regularly to develop the manual. 
Teachers with experience in curriculum writing were recommended 
for program participation by their school principals and 
selected by the district superintendent. The completed document 
was expected to correspond to requirements specified by the 
Regents Action Plan and by the Board of Education's minimal 
teaching essentials. The curriculum guide containing specific 
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lessons in the areas of basic mathematics, problem solving, and 
reasoning skills in mathematics was printed and distributed to 
all fifth grade teichers in C.S.D. 18. These teachers also 
received assistance in the use and implementation of the manual. 
EVALUATION METHODOLOGY AND FINDINGS 

The evaluation of the Mathematics Instructional Support 
System was based on a review of a completed grade five manual. 
Project staff sent the finished document to the Central Office 
of the School-Community Education Program for evaluation. It 
was determined that a curriculum specialist with the Mathematics 
Unit of the Board of Education would review the manual. A 
document review checklist was developed by the Instructional 
Support Evaluation Unit of the Office of Research, Evaluation, 
and Assessment (see Appendix A) . 

The manual was evaluated as meeting all criteria in the 
document review checklist. No written commentary was provided 
to specify the degree to which the document met the criteria for 
success, and no recommendations for improvement were suggested. 
CONCLUSIONS AND RECOMMENDATIONS 

The Mathematics Instructional Support System was a 
successful program in 1988-89. It produced a manual for fifth 
grade teachers designed to help them improve the mathematical 
abilities of their students. Overall, the manual received a 
positive rating, and its content conformed with the stated 
objective. No specific comments or recommendations for 



improvement were suggested by this reviewer. 

The document review checklist was revised for 1988-89, to 
provide a more systematic and detailed evaluation, encouraging 
written comments on e?,ch of the nine criteria. The present 
reviewer merely stated whether the manual met each of the nine 
criteria, without further elaboration. It is recommended that 
in the future, reviewers of the manual be required, not just 
encouraged, to provide written comments for aach of the nine 
criteria. This would enable a more complete evaluation of the 
strengths and weaknesses of the manual. 



Citywide Uitrella Program APFENDIX A 

Evaluation Report for Curriculum Projects Manuals 93430 
and Other Documents, (1988-89) 



Umbrella Program Name: Date: 

Name of ^i^^^n Cdnpleting the Review: 

Title: 



Introduction 



The State Education Department -requires that all Urfcrella Programs 
be evaluateed. In order to help us oeet this requirement, we are asking 
that you examine this document, and evaluate it using this form. Thank you 
for your cooperation. 



1* Itie manual follows the y n ^ 

New York State syllabus and 
the New York City curriculum. 



Explain: 



2. Ihe manual includes information Y N NA 

and requirements indicated by the 
Regents Action Plan. 



Explain: 



3. The manual integrates reasoning/ Y N NA 

thinking skills activities. 



Explain:^ 



4. Ihe manual contains lesson plans Y N NA 

that present suitable strategies 
for achieving reasonable goals. 

Expla in : 
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5. rt\e manual contains objectives and 
concepts that are clearly defined. 



Explain: 



6. Itie nanual contains classrooni 
activities and naterials that 
are relevant and consistent with 
the stated objectives and teaching 
strategies. 



Explain: 



7, Ihe manual contains criterion 
referenced tests that include 
higher-level tninking questions. 

Explain: 



The iranual contains technical 

and non-technical language that it, ^ 

consistent with the highest standaros 

of the Office of Professional Developinent 

ana Leacersnip Training, 



Expiain;^ 

9; The nanual coulc be circulatea cicywiae^ Y 

Explainj^ 



10 Ihe iranual rneetc the goals specified in 
the objective of the original proposal. 

Expla in j 



If 026 50 
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ADVENTURES IN SCIENCE, 1988-89 

School -Community Education Program 
Program Administrator: M. Morris Speiser 
Project Coordinator: Rose Viliani 

Prepared by: 
Office of Research, Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Adventures in Science prc'ect was designed to provide an 
effective science training program for teachers and supervisors 
in community School District (C.S.D.) 8. By recognizing the need 
for improved science instruction, the project trained and 
assisted participants in the davelopment and implementation of 
appropriate science lessons and activities for use by fourth and 
fifth grade teachers. These lesson plans and activities sought 
to emphasize student experimentation, problem-solving, and 
reasoning/ thinking skills. The New York State Legislature 
provided $21,000 in funding to support teacher participation in 
after-school activities. 

In 1988-89, 25 school teachers and supervisors participated 
in the project. School principals selected participants from 
among teachers who expressed the need for assistance in science 
instruct J. on and showed interest in and the ability to assi&t in 
the development of science lessons plans. Project activities, 
carried out in school and after-school worksh >s, were conducted 
by C.S.D. 8 staff and consultants. The training program provided 
a ^road overview of the New York City science curriculum and t:ie 
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New York State syllabus, it also included teaching strategies, 
laboratory techniques, and materials and activities described in 
the new Regents Action Plan, The topics discussed in the 
training sessions served as the basis for the developnent of 
lesson plans. Teachers received further on-site assistance in 
the selection and use of classroom materials and the organization 
Of classroom science centers, and through demonstration lessons. 

The project objective was for participating teachers to 
produce lesson plans appropriate for use in science by third 
through sixth grade teachers. These lesson plans had to include 
requirements indicated by the New York State Regents, the State 
Education Department, and the New York city Board of Education. 

EVALUATION METHQDQTr^G-Y AND FINDINGS 

The evaluation of the Adventures in Science program ^s 
based on a review of the completed documents through the use of 
an evaluation checklist (see Appendix A) designed to establish 
the effectiveness of the documents and determine if they met the 
requirements of the New York State Regents, the State Education 
Department, and the New York City Board of Education. 

The manual consisted of student -directed problem-solving 
science lessons designed to provide laboratory type, hands-on 
activities for use by grades 3 through 6. The completed 
documents were reviewed by a curriculum specialist from the 
Science unit of the New York City Board of Education. The 
reviewer rated all aspects of the documents very positively. No 
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Citywide Urtorella Program 
Evaluation Report for Curriculum Projects Manuals 
and Other Documents^ (1988-89) 



APPENDIX A 
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Umbrella Program Name: 



Date: 



Name of ftecia»n Completing the Review-: 
Title: 



Introduction 

Ihe State Education Department requires that all Urrtorella Programs 
be evaluateed. in order to help us meet this requirement, we are asking 
that you examine this document , and evaluate it using this form. Ihank you 
for your cooperation. 



1. Ihe manual follows the 

New York State syllabus and 
the New York City curriculum. 



N 



NA 



Explain: 



2. The manual includes information 
and requirements indicated by the 
Regents Action Plan. 



N 



NA 



Explain: 



3. Ihe manual integrates reasoning/ 
thinking skills activities. 



N 



NA 



Explain: 



4. Ihe manual contains lesson plans 
that present suitable strategies 
for achieving reasonable goals. 

Expla in : 



N 



NA 
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Umbrella Program Name: 



Date: 



Name of ftecia»n Completing the Review-: 
Title: 



Introduction 

Ihe State Education Department requires that all Urrtorella Programs 
be evaluateed. in order to help us meet this requirement, we are asking 
that you examine this document , and evaluate it using this form. Ihank you 
for your cooperation. 



1. Ihe manual follows the 

New York State syllabus and 
the New York City curriculum. 



N 



NA 



Explain: 



2. The manual includes information 
and requirements indicated by the 
Regents Action Plan. 



N 



NA 



Explain: 



3. Ihe manual integrates reasoning/ 
thinking skills activities. 



N 



NA 



Explain: 



4. Ihe manual contains lesson plans 
that present suitable strategies 
for achieving reasonable goals. 

Expla in : 



N 



NA 
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5 The manual contains objectives and 
concepts that are clearly defineo. 



Explain: 



6. The manual contains classroom 
activities and materials that 
are relevant and consistent with 
the stated objectives and teaching 
strategies. 



Explain 




7. 



The iranual contains criterion 
referenced tests that include 
higher-level thinking questions 



Explain: 



8. 



The manual contains technical 

and non-technical language that is 

consistent witl the highest stanoaras 

of the Office of Professional Development 

ana Leacersnip Training. 



Explain: 



— Tne manual coulo be circulatea citywide. Y 

Explain :_ 



10. 



The manual meets the goals specified in 
the objective of the original proposal. 



Explain :_ 
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EARLY CHILDHOOD LANGUAGE AND LITERACY PROGRAM, 1988-89 

School-Community Education Program^ 
Program Administrator: M. Morris Speiser 
Project Coordinator: Eileen Mautschke 

Prepared by: 
Office of Research, Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Early Childhood Language and Literacy project is 
designed to provide training in communication arts to 
kindergarten through grade four teachers in Community School 
Districts (C.S.D.) 9, 14, 17, 24, 28, and 29. The purpose of the 
project is to teach participants the necessary techniques and 
strategies to actively engage pupils in a structured program in 
order to improve their listening, reading, and thinking skills. 
In 1987, the program was presented the International Reading 
Association Conference in Anaheim, California. Another project 
component involved C.S.D. 14 staff in the development of 
curriculum materials in coir:nunication arts. 

Schools were selected for participation in the project 
according to their needs to improve pupil achievement in 
communication arts, and project participants were selected among 
volunteer teachers. In C.S.D. 14, participants were selected 
among staff members who had experience in curriculum development. 

The project objective was for 80 percent of the teacher 
participants to achieve an increase of at least 10 percent in 
their ability \:o teach communication arts that include listening. 
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speaking, reading, and writing in classes from kindergarten 
through grade four. Teacher performance was measured by a 
program-developed survey that was administered at the beginning 
and at the end of project activities. 

Staff members consisted of a project director and one 
teacher-trainer consultant who visited the schools and classrooms 
twice a week to provide project services. These included 
demonstration lessons, workshops, and articulation of program 
procedures. Teachers were shown how to organize their classrooms 
so that there were reading corners, listening centers, art areas, 
writing centers, and language development^'game areas. These 
areas could be used for whole group, small group, and individual 
pupil activities. Specially designed materials such as language 
development games, big books, a library of books for individual 
selection and audio-cassettes for student practice in listening 
skills were used i:i the classrooms. The New York State 
Legislature contributed $74 thousand to pay for the consultant's 
services and to purchase educational supplies. 
E VALUATION METHODOLOGY 

Project impact was assessed by an analysis teachers' 
scores on a project-developed tes'^ (see Appendix A) . The test 
was adrinistered on a pretest a:id posttest basis at the beginning 
and end of project activities. Perfect r<aw score on the test was 
75 points. In addition, the completed documents produced by the 
teachers in C.S.D. 14 were reviewed by a curriculum specialist 

2 
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through the use on an evaluation checklist (see Appendix B) . The 
evaluation checklist was designed to establish the effectiveness 
of the documents and to determine if they met the requirements of 
the New York State Regents, the State Education Department, and 
the New York City Board of Education. 
FINDINGS 

Complete test scores were submitted for 60 participants. 
Table 1 presents teacher test outcomes by C.S.D. Overall, the 
mean preprogram raw score was 32.1 points (39.7 percent correct 
responses), and the mean postprogram raw score was 46.3 points 
(61.7 percent correct), for a mean gain of 14.2 points, or 18. 
percent. Teachers at C.S.D. 9 achieved the largest mean gain of 
15.6 points, or 20.8 percent. 

Table 2 displays the percentage of teachers who achieved at 
least a ten percent increase in their ability to teach 
communication arts. Overall, 85 percent of participants met the 
project-set criterion for success. 

Teachers in C.S.l). 14 produced a written reference manual of 
reading skills that included six instructional components: 
phonics, structural analysis, vocabulary, basal reading, 
comprehension, and practice with Cloze materials. The manual was 
not found to meet any of the ten criteria of the document review 
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TABLE 1 

Teachers' Mean Raw Scores on a Project-Developed Inventory,* 

by District 

Early Childhood Language and Literacy, 1988-89 



Pretest Mean Posttest M^an Mean Gain 

Raw Percent Raw Percent Raw Percent 
N Score Correct Score Correct Score Correct 



9 


51 


29.8 


39.7% 


45.4 


60.5% 


15.6 


20.8% 


29 


9 


44.9 


59.7 


51.2 


68.3 


6.3 


8.4 


TOTAL 


60 


32.1 


42.8 


46. 3 


61.7 


14.2 


18.9 



^Perfect raw score=7 5. 

Teachers achieved an overall mean gain of 18.9 percent. 
Teachers at C.S.D. 9 achieved the highest mean gain. 
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TABLE 2 

Percentage of Teeichers Meeting the Project-Set Criterion* 
Early Childhood Language and Literacy, 1988-89 









Meetina 


cpiteripn 


C.S.D. 


N 


N 


% 


9 


51 


48 


94 .1% 


29 


9 


3 


33.3 


TOTAL 


60 


51 


85.0 



^Eighty percent of participants will achieve at least a ten 
percent increase in their ability to teach communication arts. 

Overall, 85 percent of participating teachers met the 
project-set criterion for success. 
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checklist. The reviewer stated that the manual did not meet the 
goals specified in the objective of the original proposal, or the 
requirements indicated by the Regents Action plan. She felt that 
it was not a manual per se but rather isolated sets of 
instructional skills. 
CONCLUSIONS AND RECOMMEKDATTONfi 

In 1988-89, the Early Childhood Language and Literacy 
program was successful in meeting its objective for the training 
component, and not successful in the written documents component. 

In the training component of the program, 85 percent of 
participating teachers met or surpassed the project-set criterion 
for success. C.S.D. 9 had the largest success rate, 94 percent. 
The test currently in use measures teachers' factual knowledge of 
communication arts. As a result, although the project was judged 
to be successful, it is impossible to know whether this knowledge 
is indeed being translated into improved communication 
instruction as stated in the objective. In the future, project 
staff should replace the current evaluation instrument with an 
observational survey to determine if the instruction provided in 
the program is being implemented in the classroom by 
participating teachers. 

In the second component of the program, teachers at C.S.D. 
14 were not successful in producing a manual that included 
requirements from the New York Sta'-e Regents, the State Education 
Department, and the New York City Board of Education. There 

6 
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appeared to be a discrepancy between the stated objective and the 
goals of the document produced. The objective stated chat 
teachers were to produce written instructional activities and 
lessons emphasizing comprehension skills, phonics, class 
management, and vocabulary development. The manual produced was 
described as a reference in reading skills. The reviewer stated 
that the manual appeared to be isolated sets of instructional 
skills, and lacked strategies for iii.pl ementing these skills. It 
is recommended that future manuals follow the New York State 
syllabus and New York City curriculum as closely as possible, and 
attempt to fulfill the requirements indicated by the Regents 
Action Plan. 
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Early Chi±lcilnooci Leme'vieigre Ar*t:s 



Na»e: ^ Date: 



Grade: School: 



District: 



Please indicate your answer for each of questions J-i6 by 
encircling the letter in front of one of the five oDtions 

: I 



1 On entry to school the nost Important thing for clilldren to be trained to do Is: 
a to be silent until they are asked to speak 
b to sit still at their own desks 

c to use only materials assigned to them by the teacher 
d to speak softly to other children at all tines 
e to line up quietly when asked by the teacher 



2 At tha start of the school day the teacher should first: 
a involve the children in taking attendance 
b assign the children the day's first learning tasks 
c discuss weather and news with the children 
d help the children to establish the date 
e share a new book with the children 



3 What is the best form of grouping for the teaching of reading? 
a the whole class 
b three groups based on ability 
c small mixed-ability groups 
d a combination of (a) and (c) 
e a combination of (a) and (b) 



4 What is the most important prerequisite for beginning reading instruction? 
a knowledge of letter names 
b ability to compare initial sounds 
c auditory discrimination ability 
d development of appropriate sight-word vocabulary 
e positive response to story books 



Early Childhood Language Arts Questionnaire 



93432 



0 What should be the teacher's first goal In the teachx.ig of beginning writing? 
a to develop letter formation skills 
b to Improve smEdl-muscle skills 
c to teach the purpose of writini^ 
d to extend vocabulary 
e to develop letter-sound correspondences 



6 Whac Is the Most Important consideration for the tercher when organizing the 
children's work places In the classroom? 
a children aui^ see the chsdktoard 
b children can easily talk to each other 
c children are clearly visible from the front of the room 
d children have their own designated s?&ts 
e children are not seated facing a window 



When is the best time to start teaching children to read? 
a then they can take an Interest in written language 
b when they can concentrate and follow instructions 
c when they have mastcfred basic pre-reading skills 
d when they have reached a mental ^ge of Sk years 
e when they are at the operational stage of cognitive development 



8 What is the most valuable aim for using small group games in the classroom? 

a vocabuieury extension 

b small-muscle development 

c behavior modification 

d eye-hand ccordinatiou 

e social development 



9 At the b aning of the school year» classroom work should concentrate on: 
a listening and speaking 
b listening, speaking £ind reading 
c listening, speakin,?, reading and writing 
d listening, thinkin*/ ;peaking. reading and writing 
e listening, speakii.., and writing 
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10 What should the teacher do about children's reading errors? 

a ignore them, so as not to damage the child*s confidence 

b corrffct errors on the most common words only 

c corre'jt errors that Interfsre with understanding 

d correct edl errors as they occur 

e correct errors on words previously taught 



During the eai*^- childhood school day the teacher should concentrate mainly 
a monitoring peer group interaction 
b teaching small groups of chil('""en 
c organizing whole class activities 
d supervising individual children's work 
e training children in good school behavior 



12 What is th- mo^t effective way oi promoting children's spoken language 
development? 

a teaching an extended basic vocabulary 

helping children to improve their pronunciation 
organizing spoken language interaction in small groups 
discussing appropriate topics with the class 
engaging children in in-to-one conversation 



b 
c 
d 
e 



13 For the teaching of reading in the early childhood classroom, the most important 
resource is: 

pre-primers from one well chosen basal reading program 
stoiy books with multiple cneing systems 
a wide variety of books with vs^ry simple vocabulary 
well written teacher^-made experience charts 
pre-primer from more than one basal reading program 



a 
b 
c 
d 
e 



14 Children should be encouraged to express their ideas in writing: 
a as soon as they have listened to some stories in school 
as soon as they have learned to write some letters 
as soon as they can learn to draw shapes 
as soon as they can read some words 
as soon as they enter school 



b 
c 
d 
e 
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15 Which classroom teaching approach is best for begini\ing reading? 
a a phonics approach 
b a whole-^word approach 
c a combined phonics and whole-word approach 
d a language-experience approach 
c a whole-language approach 
Give reason(s) for your answer: 



16 Please describe briefly the* kinds of staff development activity that wouji 
your opinion, be most valuable in helping you to further inprove language ana 
literacy learning in your clMsrooro. 



• 1988,^ RDR ' Reading Development Resources Ltd 
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5. 



The manual contains objectives and 
concepts that are clearly defined. 



E>cplain:^ 



6. Itie manual cent; ins classroom 
activities and naterials that 
are relevant and consistent with 
the stated objectives and teaching 
strategies. 



Explain:^ 



7. The manual contains criterion 
referenced tests that include 
higher-level thinking questions. 



Exp-lain 




8. The manual contains technical 

and non-technical language that is ^ 
consistent with the highest stanaaras 
of the Office of Professional Develoi^nt 
and Leacersnip Training. 

Explain: 



9 ^ Tne manual coula be circulateo citywiae , 
Explain:^ 



10. Ihe manual meets the goals specified in 
the objective of the original proposal. 

Explain:^^ — 
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5 The manual cx»ntains objectives and 
concepts that are clearly defined- 



E>cplain: 



N 



NA 



6. Itie manual cent; ins classroom 
activities and naterials that 
are relevant and consistent with 
the stated objectives and teaching 
strategies- 



Explain 



NA 




Ihe manual contains criterion 
referenced tests that include 
higher-level thinking questions 



Exp-lain 




8. 



Ihe manual contains technical 
and non-technical language that is ^ 
consistent with the highest stanaarcs 
of the Office of Professional DeveloFrnent 
and Leacersnip Training. 



Explain 




Tne manual coula be circulated citywiae- Y 

Explain; . 



10. The manual meets the goals specified in 
the objective of the original proposal. 

Explain:^^ ^ — 
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DISCOVERING ABILITIES AND IMPROVING ACHIEVEMENT, 1988-89 

School-Community Education Program 
Program Administrator: M. Morris Speiser 
Project Coordinator: C. Raseh Nagi 

Prepared by: 
Office of Research; Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

The Discovering Abilities and Improving Achievement program 
is designed to train teachers in Community School District 
(C.S.D.) 22 to diagnose pupil abilities and prescribe appropriate 
educational activities. This enables teachers to work with both 
gifted and talented pupils as well as with those in need of 
remedial instruction. In 1988-89, the project served 174 
elementary school teachers who were selected by school principals 
from those willing to participate in the program. 

The project coordinator and consultants conducted all-iay 
training workshops during September 1988. The training design 
was based on the Structure of Intellect (SOI) model, developed by 
Dr. J. P. Guilford and enhanced by Dr. Mary Meeker, which focuses 
on the diagnosis of student abilities and the development of 
individual prescriptive learning activities. Teachers were 
trained to develop students' cognitive skills, to differentiate 
the curriculum for potentially gifted pupils, and to implement a 
diagnostic/prescriptive classroom program. Project staff also 
assisted teachers with follow-up activities and classroom visits. 

The project objective for 1988-89 was for 80 percent of the 



participants to achieve a 40 percent increase in their ability to 
understand a diagnostic/prescriptive critical thinking program 
based on the SOI theory as measured by a project-develrped test* 
The New York State Legislature contributed $9 thousand in funding 
to cover expenses for substitute teachers. 
EVALUATION METHODOLOGY AND FINDINGS 

The evaluation of the project was based on analysis of 
teacher performance on a project-developed test consisting of 16 
multiple-choice items (see Appendix A) . Pretest and postcest 
mean raw scores were compared to determine achievement 
differences. 

Complete test data were submitted for 174 teachers. Table 1 
presents teachers' mean raw scores. Pretest mean raw score was 
10.1 points (63.1 percent correct responses) and posttest mean 
score was 13.4 points (83.8 percent correct), for a mean gain of 
3.3 points or 20.6 percent. Overall, only one percent of teacher 
participants met the project-set criterion for success of a 40 
percent increase from pre- to posttest. 
CONCLUSIONS AND RECOMMENDATIONS 

In 198 7-89, the Discovering Abilities and Improving 
Achievement project was not successful in meeting its objective 
of 80 percent of teacher participants achieving a 40 percent 
increase their ability to understand a diagnostic/prescriptive 
critical thinking program. The failure of the program to meet 
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TABLE 1 

Teachers' Mean Raw Scores* on a Project Developed Test, 
Discovering Abilities and Improving Achievement Program, 1988-89 







Pretest Mean 


PosttGst Mean 


Mean Gain 


N 


Raw 
Score 


Percent 
Correct 


Raw 
Score 


Percent 
Correct 


Raw Percent 
Score Correct 


174 


10.1 


63.1% 


13.4 


83.8% 


3.3 20.6% 



•perfect Raw Score=16. 



Overall, one percent of teacher participants met the 
project-set criterion for success a 40 percent gain 
from pretest to posttest. 
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its Objective is due in part to the ceiling effect. The ceiling 
effect occurs when a test is too easy. Many of the teachers 
scored pretest scores as high as 14 raw score points or 87.5 
percent correcc responses. Over sixty percent of teachers 
received pretest scores of over 60 percent correct responses. 
Such high pretest scores preclude a 40 percent gain on posttest. 
It is recommended that in the futute, the test be revised to 
eliminate those items teachers know before entering the program. 
In addition, the objective needs further revisic.i. The objective 
should be revised to state, "75 percent of the participating 
teachers will achieve a gain of 20 percent." 
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l^-j,^. ^^^^ Ir.tftllectual Ability is zsstsstd by these questions. 

1. Which sounds (shapes) are alike? hhich ones can be put together? 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

2. Alphabetire these words. Put the numbers that are alike together* 

a; cognition b) memory c) evaluation d) convergent producrion 

c) divergent production ^ j 

3. hhat do these words near.? (vocabulary) 

«a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

4. Do you renember which figure goes with this one? 

a) cognition b) aeaory c) e^ luation c) convergent production 
el <!iver^ent production 

5. hnat card did I just show you? (playing cards) 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

6. In the story we read, who was the sain character? fthat did he do? 
Uho was his friend? ..here was he from? Etc... 

a) cognition b) Bemory c) evaluation d) convergent production 
e) divergent production 

7. Find two objects that are related to each other, hhy are they related? 

a) cognition b) memory --c) evaluation d) convergent production 
e) diver.^ent production , >' ^ • 

8. hhich of these words are related to each other because of the way 
they are spelled? or sound? 

a) cognition b) memory c) evaluation d) convergent production 
e] divergent production 

9) K'hich words or ideas go together? V<hy?- 

a) co^ition b)mer.ory c) evaluation d) convergent production 
c) divergent production 

10} Put these pictures in order that they should go in. 

a) cognition h) r.er. Dry c) evalua:icn d) ccnve.*gent procjction 
e) divergent production > • 

11) Uhat is 1 ♦ 3? (6 - 4? S - 2? etc.) Wr.az is ? fovr letter word 
the: sizrii ult!. M irc ercis with £? 

92 
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a) copiirion b) oeaory c) evaluation d) con^'er^jenr production 
e) divergent production 

12, If you did this ?2rticular task, or used this tool, what would . 
your occupation be? 

a) co^ition b) oeaory c) evaluation d) converjent production 
e) divergent production 

13* Make something out of this clay^ paper, tile, etc. 

a) cognition b) meaory c) evaluation d) convergent production 
e) divergent production ^ 

14. Take all these noses and cars and thing's and see if you can make 
new faces • 

• 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

15. Make a nev word with the eadinr letter of this rord. Rewrite 
this song CT rhyme. 

a) cognition b) memory c) evaluation d) convergent production 
e) divergent production 

16. Can you write a poem? 

3) cognition b) meaary c)' evaluation d) convergent production 
e) divergent production 
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Project staff consisted of one teacher-coordinator. The New 
York State Legislature provided $4,600 in Junding to purchase 
instructional supplies and equipment. 

The program objectives for 1988-89 were for teacher 
participants to demonstrate a statistically significant mean 
gain on a project developed test measuring knowledge of music 
theory, and to assign a value of 15 or mote o* a survey 
measuring the impact of the workshops on their classroom 
instruction. 
EVALUATION METHODOLOGY 

Evaluation activities focused on analyses of teacher 
performance on two forms of a project-developed test (see 
Appendix A) . A different form was administered for each 
workshop series. A perfect score for each test form was 100 
points. Teachers took the test at the beginning and at the end 
of project activities. 

In addition participants completed a survey measuring the 
impact of the workshops on their classroom instruction (see 
Appendix B) . The survey asked teachers to rate five items 
assessing the impact of the workshops on areas such as their 
confidence to utili2:e and organize musical activities in their 
classroom. Teachers rated each item on a scale from 1 (poor) to 
5 (excellent) . The survey was administered at the end of the 
project activities. 
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York State Legislature provided $4,600 in Junding to purchase 
instructional supplies and equipment. 

The program objectives for 1988-89 were for teacher 
participants to demonstrate a statistically significant mean 
gain on a project developed test measuring knowledge of music 
theory, and to assign a value of 15 or mote o* a survey 
measuring the impact of the workshops on their classroom 
instruction. 
EVALUATION METHODOLOGY 

Evaluation activities focused on analyses of teacher 
performance on two forms of a project-developed test (see 
Appendix A) . A different form was administered for each 
workshop series. A perfect score for each test form was 100 
points. Teachers took the test at the beginning and at the end 
of project activities. 

In addition participants completed a survey measuring the 
impact of the workshops on their classroom instruction (see 
Appendix B) . The survey asked teachers to rate five items 
assessing the impact of the workshops on areas such as their 
confidence to utili2:e and organize musical activities in their 
classroom. Teachers rated each item on a scale from 1 (poor) to 
5 (excellent) . The survey was administered at the end of the 
project activities. 
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FINDINGS 

Complete test scores were submitted for 17 teachers who 
-attended different workshops. Six teachers attended the Staff 
P.^corder/Flutophone Workshop (SRW) , and eleven participated in 
the Music In-Service Workshop (MSW) . Table 1 shows evaluation 
findings by workshop. Mean pretest raw score was 19.5 points 
and mean posttest score was 86.5 points, for a mean gain of 67 
points. Using the paired t-test, this gain was found to be 
statistically significant. Mean gain scores varied little 
between the different workshops. 

Table 2 presents the mean ratings for the survey. A total 
of 30 teachers completed the survey forms: 11 M.S.W., 6 S.R.W., 
and 13 from a one-day workshop. All participants reported that 
the workshops had a beneficial impact on their classroom 
instruction. Total mean rating for all five items was 24 points 
out of a possible 2 5. 
CONCLUSIONS AND RECOMMENDATIONS 

In 1988-89; the Sum in One project was a successful 
program. Participants achieved significant mean gains in their 
knowledge of music theory. A compari*^ ^n of mean pretest scores, 
which were relatively low, with mean posttest scores indicates a 
remarkable improvement in teacher performance. In addition, all 
participants stated that the workshops nad a beneficial impact 
on their classroom instruction. Yet, measuring project impact 
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TABLE 1 

Teachers • Mean Gain Scores" on a Program-Developed 

Test, by Workshop 
Sum In One, 7.988-89 



Pretest Posttest Gain 

Workshop N Mean SD Mean SD Mean SD 



S.R.W.'' 


6 


10.7 


9.6 


72.0 • 


8.2 


61.3 


10.8** 


M.S.W.' 


11 


24.4 


9.1 


94.4 


5.9 


70.0 


9.7** 


TOTAL 


17 


19.5 


11.2 


86.5 


12.8 


67.0 


10.7** 



^Perfect raw score on each workshop test=100. 

^Staff Recorder/Flutophone Workshop. 

^Music In-Service Workshop. 

^Significant at p<.05. 

Teacher participants in both workshops received 
statistically significant mean gains. 



93436 



TABLE 2 

Teachers' Mean Scores^ on Project-Developed 
Survey, by Workshop 
Sum In One, 1988-89 





Workshop 


N 


Mean Ratiny 




Staff Recorder/ 
Flutophone Workshop 


6 


23.5 




Music-In-Service 
Workshop 


11 


23.2 




One-Day 
Workshop 


13 


25.0 




TOTAL 


30 


24.0 




"Perfect rating on the 


survey=25. 






* All participants rated the program 
positive impact on their classroom 


as having a 
instruction. 
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remains problematic because the tests cannot adequately measure 
the project objective or the teachers' "ability to extend and 
enrich their students' music experiences and activities." The 
tests measure factual musical knowledge but do not provide a 
measure of how this knowledge is extended to children or how it 
enriches the children's music experiences and activities. The 
survey makes an attempt to measure the projects impact on 
classroom instruction bat it is subjective in nature and deals 
primarily with the teachers perceived ability to teach music, as 
opposed to the actual impact such instruction has on the 
students. In order to effectively evaluate the improved 
instructional skills of teachers, project staff should consider' 
assessing student growth which, together with teacher 
evaluations, could provide an indirect measure to assess teacher 
instructional ability. 

In addition, the objective for the project-developed test 
should be changed. In general, t-tests of significance are 
reserved for standardized tests. As a result, it is recommended 
that the objective be changed from a "statistically significant 
increase" to "70 percent of the teachers will make a 25 percent 
increase. " 
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DAIE 



I.2.3,4.5. List 5 rea sons fsr includir^ che scudy of oum skills and si^csi::ging £a the cirriculuiu 



6.7.8.9. 10» II. Wricfi 4 C Mijor scale in boch the treble and bess clefs. (Cans noce wich letter 
. AND sol&ge syllable. 



:Ma.jar 



T 




12. 12.14.15.*. Write a fstjr aszsure rhytha la 4/4 t£=e. 
16.17.18.19. Writs a fojr ceasure rhytha in 3/4 tins 
Use the.followir^ sjttoIs: '* 



20. 21.22. 23. 24. 25. 26. 27. 2S.29. Macch Che wards and svejoIs with a csnnaccLTg line. 



ERJC 



27. eighth note 

28. dotted half note 

29. treble clef sign 

30. bass clef sicn 

31. crcscondo 

32. cLTlr.uendo 
3 J* soft 

34. Icud 

35. rccoat 

36 vhole note 
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- 2 - 

Play 1 song oc vair «n dcice that you have leazaed ia this course. 

2QJU cotts 22.22. rh.vt.hn 34.35. te» 26. tod: 37. patasiag 
S£j*icrt3d Che fiallcwing: 

38.39. Clip ±s rhytha. 
^.41. Say Ae oote oaiss. 
62.43. Siag the solSegs syllables. 
44.45. Play ca t^hs piaao cr keyboard. 
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46.47.42.49.30. Use the Plan Grid to help yoa outline deveiopLng skills « * solfege paftem of 
SSf^^""' " " """"^ "^iv-atioTpiaratiax, d^vS^'S^ °' 
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347 Baltic Scrccc 
Brooklyn. N.Y. 11201 



Of. lUesM JL GfMi 



Survey For The Umbrella Program^ 
Sum In One 53^^01 (#36) 



School ; Gradc:^^^ District 

Name: Date: 

Check if Approprirte: Special Education 

Bi 1 ingual 



Pi recti ons : Please rate the activity you participated in by putting 

a check in the box which corresponds to your assessment of the program, 
using a scale from 1-5« 



The Instrurtional Activity: 





POOR 
1 


BELOV/ 
AVERAGE 
2 


AVERAGE 
3 


ABOVE 
AVERAGE 
k 


EXCELLEKT 
5 


!• Provided guidance on how 
to organize musicci 
activities for the classroom 












2. Increased my confidence 
to use various musical 
activi ties* 












3« Helped to increase my 
musical ski 1 1s. 












k. Will enable me to integrate 
music with other subject 
areas. 












5« V/as worthwhile, and I 
would recomnend this 
program to other teachers. 













Rfiw Score 
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ORAL HISTORY PROGRAM, 1988-89 



School-Community Education Program 
Program Administrator: M. Morris Speiser 
Project Coordinator: Michael B. Gordon 

Prepared by: 
Office of Research, Evaluation, and Assessment 
Instructional Support Evaluation Unit 
New York City Public Schools 

PROJECT DESCRIPTION 

Thft Oral History Program provides training to junior high 
school teachers in Community School Districts (C.S.D.s) 9, 12, 
20, 24 and 28 in the techniques necessary for the implementation 
of an instructional program in oral history. The goal of the 
project is to complement and reinforce instructional activities 
in the areas of communication arts, social studies, and critical 
reasoning and thinking skills, and to involve students of 
participating teachers in community life. 

In 1988-89, 80 teachers and some 2,400 students 
participated in project activities. Volunteer teachers 
interested in the program were recommended for program 
participation by school principals. Teachers attended after- 
school seminars, workshopis, and in-classroom demonstration 
lessons conducted by a consultant on topics such as 
interviewing, research, questionnaire development, 
comprehension, writing, critical evaluation, and oral 
presentation. Additional activities included training in the 
use of audio-visual equipment, use of public media, advertising 
outreach techniques, and interviews with celebrities (i.e. Elie 
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Wiesel, Dizzy Gillespie, Nat Hentoff) and conununity people. 
After the training of teachers was completed, pupils 
participated in these activities. 

The project objective was for 80 percent of teacher 
participants to improve their knowledge and skills necessary to 
implement an oral history program in their classrooms. This 
improvement was expected to be at least 25 percent, as measured 
by a programs-developed test. The project was funded for $43 
thousand by the New York State Legislature. 

EVALUATION METHGDOLOGV AND FINDINGS 

Evaluation activities focused on analysis of teachers' 
scores on a project-developed test, consisting of 25 multiple- 
choice, and true and false items on interviewing techniques and 
general factual knowledge (see Appendix a) . The test was 
administered on a pretest and posttest basis at the beginning 
and end of project activities. 

Pretest and posttest scores were reported for 35 teachers 
(see Table 1). Overall, mean pretest raw score was 7.4 points 
(29.6 percent correct responses); mean posttest raw score was 
18.2 points (72.8 percent correct), for a mean gain of io.8 
points or 43.2 percent increase. All of the participating 
teachers met or surpassed the project-set criterion for success 
of a 25 percent gain at posttest. 

2 
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TABLE 1 



Teachers' Mean Raw Scores* on a Project-Developed Test, 

by District 
Oral History Program, 1988-89 







Pretest Mean 


Posttest Mean 


Mean 




N 


Raw 
Score 


Percent 
Correct 


Raw 
Score 


Percent 
Correct 


Raw 
Score 


Percent 
Correct 


35 


7.4 


29.6 


18.2 


72.8 


10.8 


43.2 



^Perfect Raw Score=25. 



Overall, mean gain was 43.2 percent. 

* One hundred percent of teachers met or surpassed the 
project-set criterion of a 25 percent gain at 
posttest. 
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CGNCLUSr ONS AND RgC0MMEWDAT TO|J<; 

In 1988-89, rhe Oral History Program was successful in 
meeting its objectj.ve. one hundred percent of participating 
teachers improved their performance at posttest by at least 25 
percent, overall, mean gain was io.8 raw score points or 43 
percent. It is recommended that alternative evaluation 
instruments be explored. The project-devfeloped test only 
measures factual knowledge and interview techniques without 
evaluating the acquisition of skills for program implementation. 
Appropriate test items measuring this ability should be included 
in the testing instrument. A survey, based on the observation of 
teacher performance in the classroom would, however, provide a 
better evaluation instrument. 
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ORAL HISTORY PROGRAM 
PRE- AKC POST- EVALUATION QUESTIONS 



To help understand the Holocaust ycu would interview: 

a. Nassau County Executive Frances Puree!! 

b. E! 1 e Wiese! 

c. Arthur Kinoy 

d. A!! of the above 

Who wrote "Return to Sender"? 

a, E!vis Presley 

b, Jimmie Lee Jones 

c, wni Dee 

d, Otis B!ackwe!l 

A good opening Oral History question is: 

a. flow o! d are you? 

b. Can you think of funny anecdotes about your family? 

c. How fTfcny kids do you have? 

d. What has your life been like? 

A pupil doing an Oral History should: 

a. Have at least tv^enty questions prepared, 

b. Prepare three to six questions, then improvise. 

c. Inprovise the interview. 

c. Know exactly what information he or she v/ants to elicit. 

A teacher preparing a class to interview their foinily nemb/^rs 
should: 

a. Write questions that students should ask on the board. 

b. Decioe what subjects to explore. 1 

c. Role play the interview. 

d. All cf the above. 

It is best to: 

a. Avoid posing questions in the interview in order to achieve 
spontanei ty . 

b. Prepare the interviewee for anything that might come up so 
there are no unanticipated problems. 

c. Pick the wording of most questions with scientific precision. 

d. Prepare the interviewee in advance for cuostions that mav be' 
posed. 
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7. J>"e or false: One must always tell the subject precisely how 
his/her memories may be used and distributed. 

S. True or false: It is best to avoid excessively emotional 
memories. 

9. True or false: The interviewee should not change the wordino of 
some questions to the subject, if (s)he does, findinos may not be 
scientifically val id. - j 

10. True or false: Conducting an Oral History will effectively 
develop some critical thinking skills. 

11. Jrue or false: Conducting Oral Histories is considered effective 
'in developing writing skills. 

12. The song "All Shook Up" was written to describe: 

a. A Coke bottle. 

b. A dance. 

An unnerving experience, 
d. A love affair. 

13. "Foxfire" is: 

a. A Boston Oral History Program. 

b. A technioue of interviewing. 

c. An Appalachian region Oral History Prooram. 

d. A folk tale first told by early American Indians. 

H. Oriana Fallacies style could be best classified as: 

a. Focused 

b. Conf ronta ti onal 

c. Probing 

d. Innovative 

e. Open-ended 

f. Approving 

g. Conversational 

15. Studs TerkeTs style could best be classified as: 

a. Focused 

b. Confrontational 

c. Probing 

d. Innovative 

e. Open-ended 

f. Approving 

g. Conversational 
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16. James Buckley^s style could be b^st classified as: 

a. Focused 

b. Conf ron cati onal 

c. Probing 

d. Innovative 

e. Open-ended 

f. Approving 

g. Conversational 

17. True or false: In a good Oral History, the role of interviewer 
and interviewee can easily reverse itself. 

18. A good edditional funding agency for a local Oral History project 
woul d be: 

a. The Ford Foundation 

b. Unitarian Universal ist Veitch FoundatioJi 

c. Villers Fund for Seniors 

d. J. K. Kaplan Fund 

19* An analogous c^efense to the Mazis* at Nuremberg is now being 
fashioned by: 

a. President Botha; South Africa Apartheid 

b. Micaraguan Sandinistas; World Court Suit 

c. Claus Von Bulow; New York Murder Trial 

d. French Government; Greenpeace Incident 

20. True or false: The use of videotape for Oral Histories, if you 
have access to it, is far superior to tape recorded material. 

21. True or false: The Oral History Program will improve the vrit^'*tg 
i n your cl ass . 

22. True or false: The Oral History Program will improve Social 
Studies instruction in your class. 

23. * True or false: The Oral History Progran will improve the 

speaking ability cf your students- 

24. True or false: The Orel History Program will improve the poise 
and self-confidence of your students. 

25. True or false: The Oral History Progran will improve the 
critical thinking capabilities of your students. 
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